Messages

Since installing 20.7 my SNMP monitoring is not working correctly. Interfaces which previously were delivering stats via SNMP are now showing zero utilisation, or only a few packets here and there.

I'm not sure if it's related to the new netstat functionality, but the os-net-snmp package isn't working well with 20.7.

Since upgrading from 20.1.7 to 20.1.8 I've seen increased latency on the WAN. I use thinkbroadband to monitor the WAN link, plus I have a ping which runs regularly from a monitoring platform on my network. After I performed the upgrade at 14:30 yesterday, my latency and jitter has gone up significantly.

Attached are two graphs. Were there any code changes or package updates that might have caused this? At this stage, I can't rule out my ISP but the timing is exactly when I performed the upgrade.

(the large red spikes in the first graph are outages when I restarted opnsense)

Not sure if this helps, but when trussing the process some errors do appear regularly every few seconds:

Code Select Expand

socket(PF_INET,SOCK_DGRAM,0) = 10 (0xa)
ioctl(10,SIOCGIFMEDIA,0x6ecaa4521270) ERR#22 'Invalid argument'
close(10) = 0 (0x0)
socket(PF_INET,SOCK_DGRAM,0) = 10 (0xa)
ioctl(10,SIOCGIFMEDIA,0x6ecaa4521270) ERR#22 'Invalid argument'
close(10) = 0 (0x0)
socket(PF_INET,SOCK_DGRAM,0) = 10 (0xa)
ioctl(10,SIOCGIFMEDIA,0x6ecaa4521270) ERR#25 'Inappropriate ioctl for device'
close(10) = 0 (0x0)
socket(PF_INET,SOCK_DGRAM,0) = 10 (0xa)
ioctl(10,SIOCGIFMEDIA,0x6ecaa4521270) ERR#25 'Inappropriate ioctl for device'
close(10) = 0 (0x0)
socket(PF_INET,SOCK_DGRAM,0) = 10 (0xa)
ioctl(10,SIOCGIFMEDIA,0x6ecaa4521270) ERR#22 'Invalid argument'
close(10) = 0 (0x0)
socket(PF_INET,SOCK_DGRAM,0) = 10 (0xa)
ioctl(10,SIOCGIFMEDIA,0x6ecaa4521270) ERR#22 'Invalid argument'
close(10) = 0 (0x0)

I've been looking at tuning some elements of my OPNsense system and whilst doing that I've noticed the the SNMP process is using over 5% of my CPU when idle (when no SNMP queries are being sent to it).

Code Select Expand

PID USERNAME       THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
67970 root             1  22    0 26004K 15188K select  2   0:17   5.04% /usr/local/sbin/snmpd -p /var/run/net_snmpd.pid

If you update 'top' manually with the space bar or set the interval to 1 second, the snmpd process spikes sometimes to 15-30% cpu when idle (no SNMP queries).

(the above was just after a reboot so the total CPU time was still low - I'll post an update once the server has been running for some time).

I don't see why SNMPD should be using quite so much CPU (for reference, this server is running on an i3-8300T).

I see a few errors in /var/log/snmpd.log but these only seem to written ever 30 seconds or so when idle:

Code Select Expand

error on subcontainer 'ia_addr' insert (-1)

When receiving an actual SNMP query (every 5 mins), quite a few of these get written to the log:

Code Select Expand

error on subcontainer 'swrun container' insert (-1)

There's very little configuration changes that can be done to Net-SNMP so was wondering if there was an issue in the implementation.

Just saw this thread which identifies the problem and a patch to solve it:

https://forum.opnsense.org/index.php?topic=16078.0

I see to have fixed this for now by SSH'ing to the router and selecting '12) Update from console' - when I did that it said it was going to remove the packages '/var/cache/pkg/sqlite-3-3.30.1,1.txz' - after doing that and rebooting the problem seems to have gone away for now, but I will continue to monitor to see if it comes back

No matter which log file I try, I can't see any log entries in the GUI. The screen always shows 'loading...' (see attached pic).

This is true which log file I choose:

Firewall / Plain View
Services / Intrusion Detection / Log File
Unbound DNS / Log File
System / Log Files / General, Backend, Web GUI
etc. etc.

I tried removing all logs in /var/log and rebooting. There are log files in /var/log with data in them.

The sqlite3 process (/usr/local/bin/sqlite3 /var/netflow/src_addr_086400.sqlite.fix) has started thrashing my disk.
The disk light is on permanently and 'top -m io -o write -s 1' reports 100% utilisation all the time:

PID USERNAME        VCSW  IVCSW   READ  WRITE  FAULT  TOTAL PERCENT COMMAND
58129 root             610      2      0    762      0    762 100.00% sqlite3

I've tried 'Reset Netflow Data' and 'Repair Netflow Data'. I've also tried deleting everything in /var/netflow and rebooting. Every time the server comes back up, this process starts thrashing the disk. The only thing I can do right now is kill -9 the process.

Any ideas on what I can do to fix this?

Thought I would post a quick update. I did a BIOS update and this is now working correctly. Seems like it was a BIOS issue (although there's nothing in the BIOS release notes about fixes in this area). FYI this was a Gigabyte ITX board.

I currently have 'Use PowerD' enabled in the GUI with all 3 settings set to Adaptive; but I don't think it's working.

From a console, I load up the CPUs and run 'powerd -v' and from the output it seems as though it wants to change the CPU frequency but it can't:

Code Select Expand


root@OPNsense:~ # powerd -v
powerd: unable to determine AC line status
load 400%, current freq  800 MHz (15), wanted freq 3200 MHz
changing clock speed from 800 MHz to 3100 MHz
load 400%, current freq  800 MHz (15), wanted freq 6200 MHz
changing clock speed from 800 MHz to 3100 MHz
load 400%, current freq  800 MHz (15), wanted freq 6200 MHz
changing clock speed from 800 MHz to 3100 MHz
load 400%, current freq  800 MHz (15), wanted freq 6200 MHz
changing clock speed from 800 MHz to 3100 MHz

Here's the output on idle:

Code Select Expand


root@OPNsense:~ # powerd -v
powerd: unable to determine AC line status
load   0%, current freq  800 MHz (15), wanted freq  800 MHz
load  42%, current freq  800 MHz (15), wanted freq  896 MHz
changing clock speed from 800 MHz to 1000 MHz
load  18%, current freq  800 MHz (15), wanted freq  868 MHz
changing clock speed from 800 MHz to 1000 MHz
load   0%, current freq  800 MHz (15), wanted freq  840 MHz
changing clock speed from 800 MHz to 1000 MHz
load   4%, current freq  800 MHz (15), wanted freq  813 MHz
changing clock speed from 800 MHz to 1000 MHz
load   0%, current freq  800 MHz (15), wanted freq  800 MHz
load  16%, current freq  800 MHz (15), wanted freq  800 MHz
load   0%, current freq  800 MHz (15), wanted freq  800 MHz


I've checked the BIOS, there's nothing obviously wrong (e.g. power management is turned on). I've also checked the CPU frequency using sysctl whilst under load repeatedly, and freq never changes:

Code Select Expand


root@OPNsense:~ # sysctl -a | grep dev.cpu.0.freq:
dev.cpu.0.freq: 800

Any ideas on getting this working? I've tried the other settings in the GUI (max,min,hiadaptive) - the behaviour does not change.

Nevermind - I found the solution.

I needed to set "Bypass firewall rules for traffic on the same interface" under Firewall -> Settings -> Advanced

I have Opnesense set up with a LAN interface (192.168.1.0/24) and a WAN Internet interface - standard NAT setup etc.

I added another new router to my LAN (192.168.30.0/24) that default gateways to Opnsense. On Opnsense I've added a new gateway for the new router, and added a static route to 192.168.30.0/24 - plus did the NAT rules etc. Outcome, hosts on 192.168.30.0/24 can access the Internet on the WAN, plus I can access the Opnsense web portal on 192.168.1.0/24.

However, hosts on 192.168.30.0/24 cannot access hosts on the 192.168.1.0/24 network. In a network trace on a .30 host, I can see packets coming in - but no packets going out (TCP connection won't establish). If I look on Opnsense, I can see in the Live Firewall logs that Opnsense is blocking the return traffic by the 'default deny rule':

   lan      Oct 31 17:50:53   192.168.1.198:22   192.168.30.12:52372   tcp   Default deny rule

The .30 host default gateway points to Opnsense. If I add a static route to the .30 host (e.g.  route add -net 192.168.30.0/24 gw 192.168.1.250 then magically it all works, and Opnsense doesn't block at the firewall level. I've tried adding in firewall rules to allow all the traffic on the LAN interface but nothing works.

My question: how do I get this to work? Why does adding a manual static route to a host magically let the traffic through the firewall?

This would be a really useful feature. We can already define schedules for firewall rules, could we look at providing the same support for the firewall rules in the traffic shaper?

There will be circumstances where traffic needs to be prioritised differently at different times of the day, e.g. work hours prioritise certain interactive traffic, evening prioritise more bulk or backup traffic etc.

I've hit on a configuration which now works - after much tweaking and reading this post - https://forum.opnsense.org/index.php?topic=7423.0

The main difference seems to be introducing a queue (as opposed to going directly to the pipe in the rules), plus switching from a WFQ or FIFO scheduler type fo FlowQeue-CoDel.

This is now working as expected on my server:

[SUM]   0.00-30.06  sec  1.35 GBytes   387 Mbits/sec              sender
[SUM]   0.00-30.06  sec  1.35 GBytes   387 Mbits/sec              receiver

The above is using 10 parallel streams. I'm posting my traffic shaper config below in case anyone else may run into the same problem. Thanks all for your assistance.

<TrafficShaper version="1.0.1">
      <pipes>
        <pipe uuid="852389a7-b347-46f5-b037-98c2d3af03fd">
          <number>10000</number>
          <enabled>1</enabled>
          <bandwidth>403</bandwidth>
          <bandwidthMetric>Mbit</bandwidthMetric>
          <queue>10</queue>
          <mask>none</mask>
          <scheduler>fq_codel</scheduler>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>0</codel_ecn_enable>
          <fqcodel_quantum>1000</fqcodel_quantum>
          <fqcodel_limit>1000</fqcodel_limit>
          <fqcodel_flows/>
          <origin>TrafficShaper</origin>
          <delay/>
          <description>down-pipe</description>
        </pipe>
        <pipe uuid="58d6c82d-bde9-4853-8fb0-d8941f38582b">
          <number>10001</number>
          <enabled>1</enabled>
          <bandwidth>23</bandwidth>
          <bandwidthMetric>Mbit</bandwidthMetric>
          <queue/>
          <mask>none</mask>
          <scheduler>fq_codel</scheduler>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>0</codel_ecn_enable>
          <fqcodel_quantum/>
          <fqcodel_limit/>
          <fqcodel_flows/>
          <origin>TrafficShaper</origin>
          <delay/>
          <description>upload-pipe</description>
        </pipe>
      </pipes>
      <queues>
        <queue uuid="cb212c8a-d208-4692-8f98-41f3dc1d1aea">
          <number>10000</number>
          <enabled>1</enabled>
          <pipe>852389a7-b347-46f5-b037-98c2d3af03fd</pipe>
          <weight>100</weight>
          <mask>none</mask>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>0</codel_ecn_enable>
          <description>main-down-q</description>
          <origin>TrafficShaper</origin>
        </queue>
        <queue uuid="e19bbd16-bb1a-4932-8eea-5814f9f70abd">
          <number>10001</number>
          <enabled>1</enabled>
          <pipe>58d6c82d-bde9-4853-8fb0-d8941f38582b</pipe>
          <weight>100</weight>
          <mask>none</mask>
          <codel_enable>0</codel_enable>
          <codel_target/>
          <codel_interval/>
          <codel_ecn_enable>0</codel_ecn_enable>
          <description>main-up-q</description>
          <origin>TrafficShaper</origin>
        </queue>
      </queues>
      <rules>
        <rule uuid="afa8b077-f5c9-4f40-ad1d-3c1d05fd8395">
          <sequence>9</sequence>
          <interface>wan</interface>
          <interface2/>
          <proto>ip</proto>
          <source>any</source>
          <source_not>0</source_not>
          <src_port>any</src_port>
          <destination>192.168.1.0/24</destination>
          <destination_not>0</destination_not>
          <dst_port>any</dst_port>
          <direction/>
          <target>cb212c8a-d208-4692-8f98-41f3dc1d1aea</target>
          <description/>
          <origin>TrafficShaper</origin>
        </rule>
        <rule uuid="22e3d03f-83ab-4c12-9e1a-99ed75e8de58">
          <sequence>10</sequence>
          <interface>wan</interface>
          <interface2/>
          <proto>ip</proto>
          <source>192.168.1.0/24</source>
          <source_not>0</source_not>
          <src_port>any</src_port>
          <destination>any</destination>
          <destination_not>0</destination_not>
          <dst_port>any</dst_port>
          <direction/>
          <target>e19bbd16-bb1a-4932-8eea-5814f9f70abd</target>
          <description/>
          <origin>TrafficShaper</origin>
        </rule>
      </rules>
    </TrafficShaper>

It's a Xen virtual machine - dmesg just reports this:

xn1: <Virtual Network Interface> at device/vif/1xn0:  on xenbusb_front0

pciconf -lv doesn't list the network interfaces. How can I find out?