Messages

256

Hardware and Performance / Re: Speedstep

« on: September 12, 2019, 01:58:24 am »

The easiest way to check speedstep is to SSH to the firewall and run 'sysctl -a dev.cpu'.

In the output, you should see something like this:

Code: [Select]

dev.cpu.0.freq_levels: 1501/0 1500/0 1400/0 1300/0 1200/0 1100/0 1000/0 900/0 800/0
dev.cpu.0.freq: 800

If the CPU frequency is lower than your max frequency, speedstep is working. You can also tell if turbo boost is support if you see your maximum frequency +1mhz. In this case, in the sample provided above, 1501/0 indicates turbo is also supported. The current CPU frequency in the sample is 800mhz, which indicates that speedstep has throttled the CPU based on load and is using a lower frequency.

Before checking all this, ensure that you have PowerD enabled. System/Settings/Misc./Power Savings/ and select HiAdaptive or Adaptive.

257

Hardware and Performance / Re: H/W Recommend for small, energy-efficient?

« on: September 10, 2019, 12:07:46 am »

Speaking of the power supply, how did you calculate the power needs and know that you could get by with just the 90W version? I'm not questioning you, rather I'm just trying to understand.

Some of this is a little bit of testing and some of this is just my own back of the napkin math.

A quick breakdown:
Asrock J3455M board - ~15W TDP processor and minimal device load (disabled soundcard, onboard NIC, etc.)
Intel Quad Port NIC - 5W max when all ports are running a 1GB/sec
120GB SATA SSD - ~1W maybe? and no wattage for spinup time unlike a hard disk

So in theory, I should see around 25W max if I somehow manage to use all the CPU and have full gigabit traffic on all of my ports and somehow manage to use a lot of I/O activity on the SSD. Yes, it COULD happen but in my use case, it's unlikely.

Now, if I run some openssl speed tests to tax the processor, I see around 16W max usage. This is measured at the outlet with a watt meter. In most cases when the CPU isn't maxed out I see 8-12W of usage, so average out it's around 9-10W consistently. Most of the time the CPU usage remains quite low.

Because of these factors I believe it's possible to support this platform with a much smaller power supply than the one I purchased. A smaller power supply may even end up being a watt or two more efficient as well.

Also on more thing regarding an ITX or mATX case. Yes, they are expensive. However you can mount an mATX board in a full size ATX case if you have a spare one laying out. This is not space efficient but if you are storing the router somewhere out of sight (like a basement), it may not matter very much. In my case I used a left over ATX case and it's been fine. If space is at a premium, then the Fitlet2 is even more compelling due to its packaging.

258

Hardware and Performance / Re: H/W Recommend for small, energy-efficient?

« on: September 09, 2019, 05:10:50 am »

Thank you! I'm liking where you're going. Could you kindly point me towards the PicoPSU you purchased? There are quite a few out there - some rather expensive it looks like - and I'd want to make sure to get the right one.

I purchased a completely overkill 120w PicoPSU here: http://www.mini-box.com/picoPSU-120-120W-power-kit
I bought it because at the time with a coupon code it was the same cost as a lesser 90W model. You could easily get away with a 90W on this system and be fine.

You'll have to weigh other costs. For instance, I already had RAM, a case, and some spare SSDs laying around. I also had an assortment of dual and quad port NICs. So for me, I made more sense to just get a board and PicoPSU and put everything together. If you have to buy all of those components, the Fitlet2 is a compelling choice because it's just plug-and-play out of the box.

Either way, the J3455 is an excellent and powerful router platform.

259

Hardware and Performance / Re: H/W Recommend for small, energy-efficient?

« on: September 09, 2019, 03:16:39 am »

Another vote for the J3455 platform. I purchase an Asrock j3455m board, a PicoPSU, and a quad port Intel NIC. The system idles around 8-10watts and runs without any fans. It also is compatible UEFI booting OPNsense.

Here's a link to the board I purchased: https://asrock.com/mb/Intel/J3455M/

260

Hardware and Performance / Re: High CPU usage: Xeon E3-1265L V2

« on: August 28, 2019, 01:47:21 am »

loader.conf.local is parsed automatically at boot. I've never had to make modifications to loader.conf.

Just create loader.conf.local and input any changes you want and reboot. You can test if the changes were applied at boot by running sysctl hw.bce and check if the values you input in loader.conf.local are now applied after a reboot.

261

Hardware and Performance / Re: High CPU usage: Xeon E3-1265L V2

« on: August 27, 2019, 02:57:14 pm »

Disabling shouldn't harm anything. Create this file: /boot/loader.conf.local

Input those two lines in the loader.conf.local file. Best practice is to NOT modify loader.conf because it will be overwritten on an upgrade. loader.conf.local will not be overwritten and your customizations will be retained during upgrades.

You can also do some Broadcom driver tuning by increasing pages and adding these lines to loader.conf.local:

sysctl hw.bce.tx_pages=8
sysctl hw.bce.rx_pages=8

You can try modifying the RX ticks and Quick Cons values too if you like. All of this is reversible (just delete the lines you don't want from loader.conf.local and reboot). The bad news is after all of this, we're running out of tuning to help with your CPU usage. So if these items don't make a significant impact we can't do much else short of trying a new NIC chipset.

262

Hardware and Performance / Re: High CPU usage: Xeon E3-1265L V2

« on: August 27, 2019, 01:56:50 am »

If you're trying to max throughput I would expect the NIC to be using MSI-X and multiple queues to spread the packet load across multiple threads. This is what the 10GB Intel adapter is doing.

Generally on BSD, Broadcom drivers don't offer the tuning available to Intel drivers. So my first recommendation would be to try this with an Intel dual port or quad port NIC.

However, we can try a few things on the Broadcom right now and see if it helps.
First run, and copy the results of these commands:

sysctl hw.bce

vmstat -i

Then, run the following commands:

sysctl hw.bce.msi_enable=0

sysctl hw.bce.tso_enable=0

Try re-running your load tests with TSO and MSI disabled for BCE, some people report that this helps (even though it's technically less efficient). Also, you can try enabling them if they are already disabled (you can tell this from running the first command above, sysctl hw.bce, if they are already at a 0 value then try changing them to 1 and test.

263

Hardware and Performance / Re: High CPU usage: Xeon E3-1265L V2

« on: August 26, 2019, 02:50:36 pm »

That's a lot of interrupt usage. It looks like the 10GB Intel card is doing alright but the Broadcom card that is in that system is only using a single interrupt (probably one of the onboard NICs?). That seems to be a least one of the bottlenecks. As you can see, Suricata is also taking a significant CPU load, and so is NTOPng.

Can you post the output of: dmesg | grep -i msi

264

Hardware and Performance / Re: High CPU usage: Xeon E3-1265L V2

« on: August 23, 2019, 05:27:16 am »

Open an SSH session while running speed tests and run "top -aSCHIP". You can watch which process(s) is eating up CPU. Depending on what that shows will determine what the next steps are to see if it can be resolved.

265

General Discussion / Re: wan vlan questions -att fiber bypass -noob questions

« on: August 16, 2019, 10:08:32 pm »

I can't answer your question regarding the ebook but, I'm glad you got the bypass working with the smart switch method.

I have moved back to ATT service and am still using the same bypass method that you linked to from about a year ago. It still works very well and is completely reliable, I keep it all on a UPS battery backup so the power is stable. I did notice that ATT has reduced their WAN DHCP lease times down to 1 hour. However the bypass has been stable for months.

266

Hardware and Performance / Re: High CPU usage: Xeon E3-1265L V2

« on: August 16, 2019, 03:35:08 am »

Suricata may be a major contributor to the CPU spikes depending on what it is configured to do. Also it looks like you're using OpenVPN? Make sure you're using AES-NI for crypto acceleration on those VPN tunnels, otherwise you can see increased CPU utilization if you're pushing 600mbits through the VPN tunnels.

Can you try a speed test with Suricata disabled and see if this impacts the CPU usage?

267

19.7 Legacy Series / Re: High CPU Usage since upgrade from 19.1.10 to 19.7

« on: July 23, 2019, 02:35:38 pm »

Just following up on my previous post to provide some extra input. I tried first just repairing netflow data, this did not have an impact in perceived performance and CPU utilization remained high. I then completely reset RRD graphs and netflow data and rebooted the device.

Unfortunately even with these steps I've seen no improvement in page load performance. I can understand that this new version may need more core processing power for NetFlow. What doesn't make sense to me is why the whole page loads are noticeably laggy and slow compared to 19.1.

268

19.7 Legacy Series / Re: High CPU Usage since upgrade from 19.1.10 to 19.7

« on: July 21, 2019, 05:52:21 pm »

Also seeing high CPU utilization after upgrading from 19.1.10 to 19.7. As shown in the thread, it appears to be Python/Netflow related.

Code: [Select]

PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME     CPU COMMAND
   11 root       155 ki31     0K    64K CPU2    2   8:16  99.15% [idle{idle: cpu2}]
   11 root       155 ki31     0K    64K CPU0    0   9:19  89.21% [idle{idle: cpu0}]
   11 root       155 ki31     0K    64K RUN     3   8:34  85.92% [idle{idle: cpu3}]
   11 root       155 ki31     0K    64K RUN     1   8:12  72.94% [idle{idle: cpu1}]
52874 root        52    0 19736K 14632K piperd  3   0:01  44.24% /usr/local/bin/python3 /usr/local/opnsense/scripts/filte


Code: [Select]

PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME     CPU COMMAND
83957 root        84    0 28848K 25344K CPU2    2   3:34  96.98% /usr/local/bin/python3 /usr/local/opnsense/scripts/netfl
   11 root       155 ki31     0K    64K RUN     1   9:54  66.54% [idle{idle: cpu1}]
   11 root       155 ki31     0K    64K CPU3    3  10:10  63.93% [idle{idle: cpu3}]
   11 root       155 ki31     0K    64K CPU0    0  11:08  51.51% [idle{idle: cpu0}]
   11 root       155 ki31     0K    64K RUN     2   9:57  42.72% [idle{idle: cpu2}]
   19 root       -16    -     0K    16K -       0   0:17  11.31% [rand_harvestq]
   12 root       -60    -     0K   544K WAIT    1   0:03   1.03% [intr{swi4: clock (0)}]
    0 root       -92    -     0K   592K -       0   0:02   0.36% [kernel{dummynet}]
36090 root        52    0 51688K 41524K accept  0   0:04   0.34% /usr/local/bin/php-cgi
40440 root        20    0  1034M  4536K CPU1    1   0:00   0.07% top -aSCHIP

I'll try resetting Netflow data and report back. I've also noticed that the web interface is noticeably laggy after the 19.7 upgrade, again probably due to the CPU utilization. This is on a bare metal install, Celeron J3455 quad core, 16GB RAM, and a 120GB SSD. Usually a very snappy system.

269

19.7 Legacy Series / upgraded from 19.1.10 to 19.7r1, NetFlow data is not working

« on: July 11, 2019, 02:34:52 pm »

Upgraded from 19.1.10 to 19.7r1. NetFlow data is no longer working. I have tried resetting NetFlow and also repairing it.

The interface totals graph continues to work and I can see activity. However the lower graph for port usage and sources is blank and does not populate connection information.

Screenshot is attached to show the issue.

270

Hardware and Performance / Re: Intel Turbo Boost

« on: July 02, 2019, 02:41:43 pm »

Late response to this thread but, this is my understanding for Turbo Boost under *BSD based OSes.

My platform is a Celeron J3455 and I have PowderD HiAdaptive enabled. When I console in to this system, I run the following command and see this output:

Code: [Select]

sysctl -a | grep dev.est

Code: [Select]

dev.est.3.freq_settings: 1501/0 1500/0 1400/0 1300/0 1200/0 1100/0 1000/0 900/0 800/0
dev.est.3.%parent: cpu3
dev.est.3.%pnpinfo:
dev.est.3.%location:
dev.est.3.%driver: est
dev.est.3.%desc: Enhanced SpeedStep Frequency Control
dev.est.2.freq_settings: 1501/0 1500/0 1400/0 1300/0 1200/0 1100/0 1000/0 900/0 800/0
dev.est.2.%parent: cpu2
dev.est.2.%pnpinfo:
dev.est.2.%location:
dev.est.2.%driver: est
dev.est.2.%desc: Enhanced SpeedStep Frequency Control
dev.est.1.freq_settings: 1501/0 1500/0 1400/0 1300/0 1200/0 1100/0 1000/0 900/0 800/0
dev.est.1.%parent: cpu1
dev.est.1.%pnpinfo:
dev.est.1.%location:
dev.est.1.%driver: est
dev.est.1.%desc: Enhanced SpeedStep Frequency Control
dev.est.0.freq_settings: 1501/0 1500/0 1400/0 1300/0 1200/0 1100/0 1000/0 900/0 800/0
dev.est.0.%parent: cpu0
dev.est.0.%pnpinfo:
dev.est.0.%location:
dev.est.0.%driver: est
dev.est.0.%desc: Enhanced SpeedStep Frequency Control
dev.est.%parent:

My understanding is that the dev.est.x.freq_settings 1501/0 indicates that the OS will use higher clock speeds above the base 1500mhz clock for this particular processor.

Now, the second question is, can we in real time see the clock speed as it increases. This is one I don't know and I'd also like to verify this first hand to make sure that the maximum overhead is being used for the processor clock speed. However for the time being, if you see a +1mhz increment listed for your processor at the max default clock speed, this should indicate turbo is enabled.