[SOLVED] Slow throughput

[dpbklyn]

Hello and thank you in advance...

We am testing OPNSense in our office before deploying live and recently we have found that our download speed gets severely limited.

Our ISP connection is 100/4, but over the last few days we have noticed that the are seeing about 3/4 until we reboot the OPNSense at which point we see our typical Speeds.  The only recent change to the OPNSense (other than the initial setup about 4 weeks ago) was that we implemented auto-backup to google drive.

Any help troubleshooting this issue would be greatly appreciated.

Thank you,

dp

[dpbklyn]

I posgted this about 30 minutes ago after a reboot of the firewall and I was getting about 90/4...I just ran a speed test again and it is 7/4...

[bartjsmit]

You can revert to a recent config from before the back up change from the OPNsense console. Download the config from the GUI first so you can get back to the current state easily.

Bart...

[dpbklyn]

Bart,

Thank you.  I shut down the auto backup and there was no change.

I am running OPNSense on an old Dell Celeron machine with oodles of Disk space and 8 Gigs of RAM.  I don't think hardware is an issue...

I have some NATting, and the requisite rules that apply to the NATting but, really there is nothing on this router that is out of the ordinary.  I also have a OPENVPN Server running, but it is RARELY connected (only when I am out of the office)

I am going to disable the NATting and see if that helps.

Thank you,

dp

[dpbklyn]

OK...

I turned off all NATing, and plugged directly into the firewall and got the same slow throughput.  When I plug directly into the modem, my speed is back to 100/4.

Just updating...

dp

[bartjsmit]

Run top in a shell and see what's hogging the CPU and memory?

Bart...

[dpbklyn]

Bart, Thank you for your continued help!

Now my Newbie-ness is going to show...

Run top in a shell and see what's hogging the CPU and memory?

huh...?

Can you point me to some documentation?

Thank you,

dp

[bartjsmit]

Sorry for my brevity ;-)

Connect to the firewall with SSH or using a local keyboard and monitor. Log in as root and select option 8 for an interactive shell. Run top (type top and hit return) and you'll see a screen like this:

last pid: 45758;  load averages:  0.17,  0.24,  0.23    up 8+03:05:32  20:43:01
44 processes:  1 running, 43 sleeping
CPU:  0.5% user,  0.0% nice,  0.1% system,  0.1% interrupt, 99.3% idle
Mem: 99M Active, 1392M Inact, 1169M Wired, 546M Buf, 5236M Free
Swap: 16G Total, 16G Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
99048 root         12  20    0  1170M   448M nanslp  3 266:23   1.96% suricata
66872 root          1  20    0 12648K  2660K bpf     2   1:41   0.03% filterlog
28834 root          1  20    0 20032K  3364K CPU1    1   0:00   0.02% top
 8511 root          1  20    0 10468K  2508K select  1   0:43   0.01% syslogd
89559 root          1  20    0  8260K  2320K select  1   0:38   0.01% apinger
69921 root          1  24    0 99944K 26748K select  2 370:39   0.01% python2.7
  861 root          1  20    0 22612K 12512K select  2   0:12   0.00% ntpd
 9698 root          1  20    0 53688K  6836K select  1   0:00   0.00% sshd
99373 root          1  20    0 57444K  9308K kqread  2   0:19   0.00% lighttpd
  607 root          1  20    0 12476K  2388K nanslp  1   0:02   0.00% cron
28815 root          1  20    0 22396K  7240K select  2   0:28   0.00% openvpn
43269 root          1  52    0  1054M  3276K wait    2   0:22   0.00% sh
67110 _flowd        1  20    0  8300K  2576K select  2   0:08   0.00% flowd
42525 root          1  20    0 44872K  8544K select  2   0:03   0.00% mpd5
85953 nobody        1  20    0  8216K  1908K sbwait  3   0:03   0.00% samplicat
67226 root          1  52    0   112M 27448K accept  3   0:03   0.00% php-cgi
  171 root          1  20    0   123M 26252K accept  1   0:02   0.00% python2.7

This gives you the list of processes, sorted by load on the system. Run 'man top' (without the quotes) to get more familiar with the options. Given that yours is a Celeron, the most likely bottleneck is the CPU.

You could get some mileage out of a server class multi-port NIC, especially the Intel ones. These will off-load a lot of work from the CPU and are generally best supported by FreeBSD (and thus by OPNsense).

Bart...

[dpbklyn]

Thank you...

For some reason I cannot connect to the firewall with ssh...a problem for another day.  I am uploading photos of the output.

Python seems to run intermittently.  Could this slowness be caused because this is a celeron with only 1 core?

[dpbklyn]

Got connected via SSH:


  PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME    WCPU COMMAND
37400 root          6  20    0   132M 78308K nanslp  25:37   2.06% suricata
85899 root          1  20    0 20032K  3568K RUN      0:00   0.07% top
31837 root          1  20    0 12648K  2388K bpf      0:29   0.02% filterlog
 7819 root          1  20    0 10468K  2496K select   0:14   0.01% syslogd
  200 root          1  20    0  9536K  5048K select   0:09   0.01% devd
42219 root          1  20    0  8260K  2220K select   0:18   0.01% apinger
32947 squid         1  20    0   199M 50908K kqread   0:07   0.01% squid
 1945 root          1  21    0 97896K 24180K select  75:11   0.01% python2.7
84433 root          1  20    0 53688K  6840K select   0:00   0.01% sshd
61861 squid         1  20    0 33764K  4532K select   0:03   0.01% pinger
54786 root          1  20    0 20564K 10456K select   0:07   0.01% ntpd
47992 dhcpd         1  20    0 22872K 13644K select   0:00   0.01% dhcpd
32958 squid         1  20    0 33764K  4420K select   0:05   0.01% pinger
45017 root          1  20    0 57444K  8868K kqread   0:05   0.00% lighttpd
23969 root          2  20    0 10568K  2228K piperd   0:01   0.00% sshlockout_p
82954 root          1  52    0  1054M  2980K wait     0:17   0.00% sh
46117 root          1  52    0   112M 27216K accept   0:07   0.00% php

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME    WCPU COMMAND
 1945 root          1  33    0 97896K 23588K select  75:16  52.27% python2.7
37400 root          6  20    0   132M 78308K RUN     25:39   2.63% suricata
 5186 root          1  20    0 20032K  3572K RUN      0:00   0.05% top
42219 root          1  20    0  8260K  2220K select   0:18   0.02% apinger
31837 root          1  20    0 12648K  2388K bpf      0:29   0.02% filterlog
  200 root          1  20    0  9536K  5048K select   0:09   0.01% devd
55862 root          1  20    0 10352K  2244K select   0:04   0.01% radvd
 7819 root          1  20    0 10468K  2496K select   0:14   0.01% syslogd
84433 root          1  20    0 53688K  6840K select   0:00   0.01% sshd
47992 dhcpd         1  20    0 22872K 13644K select   0:00   0.01% dhcpd
54786 root          1  20    0 20564K 10456K select   0:07   0.01% ntpd
61861 squid         1  20    0 33764K  4532K select   0:03   0.01% pinger
32947 squid         1  20    0   199M 50908K kqread   0:07   0.00% squid
32958 squid         1  20    0 33764K  4420K select   0:05   0.00% pinger
45017 root          1  20    0 57444K  8868K kqread   0:05   0.00% lighttpd
23969 root          2  20    0 10568K  2228K piperd   0:01   0.00% sshlockout_p
82954 root          1  52    0  1054M  2980K wait     0:17   0.

[dpbklyn]

After a reboot, while getting 90+ Mbps:

[bartjsmit]

it looks like IDS (suricata) is taking a chunk of CPU, perhaps taking a core to itself. Can you switch it off?

How many cores does your processor have? Are they real or hyper-threaded?

Bart...

[dpbklyn]

Unfortunately, this is built on an old Dell Celeron box...There is only one core.

Intel(R) Celeron(R) CPU 450 @ 2.20GHz (1 cores)

sad...

I have another box I am trying to get up and running, but the integrated NIC is not wanting to connect and it onnly has one PCI slot for ONE additional NIC.

sad...

[dpbklyn]

Incidentally, how do you determine that Suricata is the culprit by looking at the information I provided?  I just want to learn to be more self-sufficient.

Thank you,

dp

[bartjsmit]

Suricata is using 25% of CPU time with Python taking the other 75%. The latter represents all the scripts that are running, so the single suricata binary stands out. Load averages indicate the length of the queue waiting for your CPU. Ideally this should be around the 0.2 mark or lower, meaning that a process can get serviced immediately 80% of the time. Don't forget that this is cumulative for all cores. My values of 0.17 to 0.24 are across four cores, while your much higher value is for only one.

There are other threads on this forum where suricata was identified as resource intensive and using only a single core - not so much an issue in your case ;-) If you are unable to upgrade your hardware, you may need to off-load IDS to a dedicated internal server or appliance. Suricata and VPN are typically the resource hogs on OPNsense.

Having only one slot is not an problem per se. I'm using a rather minimalistic mobo and an Intel dual port PCIe 2x card (Intel 82575EB). FreeBSD works very well with the Intel NIC's.


TL:DR - IDS needs two (decent) cores

Bart...