Messages

31

Hardware and Performance / Re: Are Realtek NICs really that bad?

« on: November 11, 2019, 04:34:39 am »

Realtek NICs tend to get a bad reputation due to the native FreeBSD driver support being poor. OPNsense does have a better driver for Realtek based NICs which can help them be more viable. Also, in situations where you are using an all-in-one device with little or no expansion options, a Realtek NIC may be the only connectivity option available.

However, in systems that are expandable with a PCIe slot, it's a minimal cost to buy a used server NIC with 2 or 4 ports and an Intel chipset. This is not strictly necessary but it's nearly a guarantee that the NIC won't be an issue down the road. Is it a necessity, definitely not. But if you have expansion options it's a minimal cost that will result in better utilization.

That's my 2c on the matter anyway. For the record, I used Realtek NICs for years on *BSD based firewalls. I always switched to Intel when possible (and when cost effective).

32

19.7 Legacy Series / Re: Firewall States Dump shows closed connections as established

« on: October 05, 2019, 05:52:20 pm »

I have only a basic understand of these so unfortunately I can't speak authoritatively as to why these defaults were chosen. Also it's worth noting that I took that screenshot from the other *sense site which is still FreeBSD based.

OPNsense may use different defaults if HardenedBSD changes these, but I don't know for sure. One of the devs would probably be able to shed more light on A) are the defaults different in HardenedBSD and B) why were the default values chosen?

33

19.7 Legacy Series / Re: Firewall States Dump shows closed connections as established

« on: October 05, 2019, 02:02:43 am »

Firewall/Settings/Advanced/Firewall Optimization is what you're looking for. Default is "normal". Be careful though, it changes more than just the parameters for "established" sessions. Attached a screenshot of option details.

34

General Discussion / Re: How to update from 16.1

« on: October 05, 2019, 01:56:08 am »

16.1 is very old. I'm not sure there is a direct upgrade path? I think your best bet would be to make a config backup and restore it on a new install. One of the devs may have another way to manually update the repos to potentially allow you to upgrade but unless it can skip packages we're talking years of versions that have come and gone.

35

19.7 Legacy Series / Re: constant ~50% cpu usage according to 'sysctl dev.cpu.*'

« on: September 12, 2019, 02:04:11 am »

Do you have the outputs of these two?

dev.cpu.0.cx_supported
dev.cpu.0.cx_lowest

If the supported value is C1 this may be why you're seeing such high values. If you have C states enabled in the BIOS and you are using the Power Savings settings, you should see C2/C3 or even higher values supported.

36

Hardware and Performance / Re: Speedstep

« on: September 12, 2019, 01:58:24 am »

The easiest way to check speedstep is to SSH to the firewall and run 'sysctl -a dev.cpu'.

In the output, you should see something like this:

Code: [Select]

dev.cpu.0.freq_levels: 1501/0 1500/0 1400/0 1300/0 1200/0 1100/0 1000/0 900/0 800/0
dev.cpu.0.freq: 800

If the CPU frequency is lower than your max frequency, speedstep is working. You can also tell if turbo boost is support if you see your maximum frequency +1mhz. In this case, in the sample provided above, 1501/0 indicates turbo is also supported. The current CPU frequency in the sample is 800mhz, which indicates that speedstep has throttled the CPU based on load and is using a lower frequency.

Before checking all this, ensure that you have PowerD enabled. System/Settings/Misc./Power Savings/ and select HiAdaptive or Adaptive.

37

Hardware and Performance / Re: H/W Recommend for small, energy-efficient?

« on: September 10, 2019, 12:07:46 am »

Speaking of the power supply, how did you calculate the power needs and know that you could get by with just the 90W version? I'm not questioning you, rather I'm just trying to understand.

Some of this is a little bit of testing and some of this is just my own back of the napkin math.

A quick breakdown:
Asrock J3455M board - ~15W TDP processor and minimal device load (disabled soundcard, onboard NIC, etc.)
Intel Quad Port NIC - 5W max when all ports are running a 1GB/sec
120GB SATA SSD - ~1W maybe? and no wattage for spinup time unlike a hard disk

So in theory, I should see around 25W max if I somehow manage to use all the CPU and have full gigabit traffic on all of my ports and somehow manage to use a lot of I/O activity on the SSD. Yes, it COULD happen but in my use case, it's unlikely.

Now, if I run some openssl speed tests to tax the processor, I see around 16W max usage. This is measured at the outlet with a watt meter. In most cases when the CPU isn't maxed out I see 8-12W of usage, so average out it's around 9-10W consistently. Most of the time the CPU usage remains quite low.

Because of these factors I believe it's possible to support this platform with a much smaller power supply than the one I purchased. A smaller power supply may even end up being a watt or two more efficient as well.

Also on more thing regarding an ITX or mATX case. Yes, they are expensive. However you can mount an mATX board in a full size ATX case if you have a spare one laying out. This is not space efficient but if you are storing the router somewhere out of sight (like a basement), it may not matter very much. In my case I used a left over ATX case and it's been fine. If space is at a premium, then the Fitlet2 is even more compelling due to its packaging.

38

Hardware and Performance / Re: H/W Recommend for small, energy-efficient?

« on: September 09, 2019, 05:10:50 am »

Thank you! I'm liking where you're going. Could you kindly point me towards the PicoPSU you purchased? There are quite a few out there - some rather expensive it looks like - and I'd want to make sure to get the right one.

I purchased a completely overkill 120w PicoPSU here: http://www.mini-box.com/picoPSU-120-120W-power-kit
I bought it because at the time with a coupon code it was the same cost as a lesser 90W model. You could easily get away with a 90W on this system and be fine.

You'll have to weigh other costs. For instance, I already had RAM, a case, and some spare SSDs laying around. I also had an assortment of dual and quad port NICs. So for me, I made more sense to just get a board and PicoPSU and put everything together. If you have to buy all of those components, the Fitlet2 is a compelling choice because it's just plug-and-play out of the box.

Either way, the J3455 is an excellent and powerful router platform.

39

Hardware and Performance / Re: H/W Recommend for small, energy-efficient?

« on: September 09, 2019, 03:16:39 am »

Another vote for the J3455 platform. I purchase an Asrock j3455m board, a PicoPSU, and a quad port Intel NIC. The system idles around 8-10watts and runs without any fans. It also is compatible UEFI booting OPNsense.

Here's a link to the board I purchased: https://asrock.com/mb/Intel/J3455M/

40

Hardware and Performance / Re: High CPU usage: Xeon E3-1265L V2

« on: August 28, 2019, 01:47:21 am »

loader.conf.local is parsed automatically at boot. I've never had to make modifications to loader.conf.

Just create loader.conf.local and input any changes you want and reboot. You can test if the changes were applied at boot by running sysctl hw.bce and check if the values you input in loader.conf.local are now applied after a reboot.

41

Hardware and Performance / Re: High CPU usage: Xeon E3-1265L V2

« on: August 27, 2019, 02:57:14 pm »

Disabling shouldn't harm anything. Create this file: /boot/loader.conf.local

Input those two lines in the loader.conf.local file. Best practice is to NOT modify loader.conf because it will be overwritten on an upgrade. loader.conf.local will not be overwritten and your customizations will be retained during upgrades.

You can also do some Broadcom driver tuning by increasing pages and adding these lines to loader.conf.local:

sysctl hw.bce.tx_pages=8
sysctl hw.bce.rx_pages=8

You can try modifying the RX ticks and Quick Cons values too if you like. All of this is reversible (just delete the lines you don't want from loader.conf.local and reboot). The bad news is after all of this, we're running out of tuning to help with your CPU usage. So if these items don't make a significant impact we can't do much else short of trying a new NIC chipset.

42

Hardware and Performance / Re: High CPU usage: Xeon E3-1265L V2

« on: August 27, 2019, 01:56:50 am »

If you're trying to max throughput I would expect the NIC to be using MSI-X and multiple queues to spread the packet load across multiple threads. This is what the 10GB Intel adapter is doing.

Generally on BSD, Broadcom drivers don't offer the tuning available to Intel drivers. So my first recommendation would be to try this with an Intel dual port or quad port NIC.

However, we can try a few things on the Broadcom right now and see if it helps.
First run, and copy the results of these commands:

sysctl hw.bce

vmstat -i

Then, run the following commands:

sysctl hw.bce.msi_enable=0

sysctl hw.bce.tso_enable=0

Try re-running your load tests with TSO and MSI disabled for BCE, some people report that this helps (even though it's technically less efficient). Also, you can try enabling them if they are already disabled (you can tell this from running the first command above, sysctl hw.bce, if they are already at a 0 value then try changing them to 1 and test.

43

Hardware and Performance / Re: High CPU usage: Xeon E3-1265L V2

« on: August 26, 2019, 02:50:36 pm »

That's a lot of interrupt usage. It looks like the 10GB Intel card is doing alright but the Broadcom card that is in that system is only using a single interrupt (probably one of the onboard NICs?). That seems to be a least one of the bottlenecks. As you can see, Suricata is also taking a significant CPU load, and so is NTOPng.

Can you post the output of: dmesg | grep -i msi

44

Hardware and Performance / Re: High CPU usage: Xeon E3-1265L V2

« on: August 23, 2019, 05:27:16 am »

Open an SSH session while running speed tests and run "top -aSCHIP". You can watch which process(s) is eating up CPU. Depending on what that shows will determine what the next steps are to see if it can be resolved.

45

General Discussion / Re: wan vlan questions -att fiber bypass -noob questions

« on: August 16, 2019, 10:08:32 pm »

I can't answer your question regarding the ebook but, I'm glad you got the bypass working with the smart switch method.

I have moved back to ATT service and am still using the same bypass method that you linked to from about a year ago. It still works very well and is completely reliable, I keep it all on a UPS battery backup so the power is stable. I did notice that ATT has reduced their WAN DHCP lease times down to 1 hour. However the bypass has been stable for months.