Messages

Right, just worked that out. For some reason, that didn't run so I just did it manually and I don't need the UI stuff anyway so removed it.

There is a plugin os-speedtest-community in the mimugmail repository which includes the CLI version.

Is this still true? I installed this on 25.1 but can't work out what the speed test executable actually is.

Code Select Expand

# pkg info -lx os-speedtest-community
os-speedtest-community-0.9_6:
        /usr/local/opnsense/mvc/app/controllers/OPNsense/Speedtest/Api/DownloadController.php
        /usr/local/opnsense/mvc/app/controllers/OPNsense/Speedtest/Api/ServiceController.php
        /usr/local/opnsense/mvc/app/controllers/OPNsense/Speedtest/IndexController.php
        /usr/local/opnsense/mvc/app/models/OPNsense/Speedtest/ACL/ACL.xml
        /usr/local/opnsense/mvc/app/models/OPNsense/Speedtest/Menu/Menu.xml
        /usr/local/opnsense/mvc/app/models/OPNsense/Speedtest/Speedtest.php
        /usr/local/opnsense/mvc/app/models/OPNsense/Speedtest/Speedtest.xml
        /usr/local/opnsense/mvc/app/views/OPNsense/Speedtest/index.volt
        /usr/local/opnsense/scripts/OPNsense/speedtest/install_speedtest.sh
        /usr/local/opnsense/scripts/OPNsense/speedtest/opn_speedtest.py
        /usr/local/opnsense/service/conf/actions.d/actions_speedtest.conf
        /usr/local/opnsense/version/speedtest-community
        /usr/local/opnsense/www/js/widgets/Metadata/Speedtest.xml
        /usr/local/opnsense/www/js/widgets/Speedtest.js
        /usr/local/www/widgets/include/speedtest.inc
        /usr/local/www/widgets/widgets/speedtest.widget.php

I was under the impression that this functionality was deprecated by Google, so have changed my backup method to Oxidized.

OK.....not the fault of the edge Opnsense firewall...the issue was the downstream L3 router, just routing the replies coming back from the internet, back out to the edge firewall again. Once I told it to just route things to the correct places, it's now working.
No wonder OpnSense and I were very confused!

You wouldn't happen to have synproxy enabled on the pass rule for these sessions?

I have tried every option under state type on this rule, none of them work.

tcpflags on the dropped packets are 'SA'.
ICMP ping has the same issue.
If I click on the rule ID in the info popup, it opens a tab which then immediately closes itself.

The state table looks "reasonable" to me - the only oddity is that the matching rule is alleged to be "let anything out of the firewall itself", which this traffic is not. But I compared it to another firewall and that's the same matching rule name.

No, there is a single WAN gateway and a single gateway to reach these routed subnets on TNET.

Scenario: on TNET interface we have a L3 routed network, one example network is 10.10.45.0/24. Routes to these networks are configured in System: Routes: Configuration.

Outbound NAT policies for these networks have been created manually, I assume they're getting hits because at the far end I can see my test traffic originating from the expected IP.

Here's the problem - the replies are being dropped because of "Default deny / state violation rule". My guess here is that the firewall knows the matching state for these rules, otherwise it wouldn't know the private IP they're for. So what is the real issue?
Devices in the TNET subnet work fine, it's just the routed subnets that don't work.

Code Select Expand

TNET        2025-06-17T08:42:25    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:42:17    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:42:09    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:42:05    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:42:01    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:41:59    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:41:57    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:41:56    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:41:55    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:41:54    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
WAN        2025-06-17T08:41:54    45.76.130.220:14043    91.2.119.28:80    tcp    let out anything from firewall host itself (force gw)   

I turned the log level back up to DEBUG4 but it's no longer logging these ipdr messages. So I will leave this one for now.

Bug reported.
I notice that in the current version of Zenarmor [vs whatever version it was in July 2024], it doesn't attempt to redirect output at all.

This still happens with current versions of OpnSense + Zenarmor. Simply redirecting the output of "zenarmor peridiocals" to /dev/null would fix this.

Three times in past 4 weeks, disk has filled up on my OpnSense and every time it has been something to do with Zenarmor.
First one was cron jobs that fail to send an error message once per minute. Fixed that.
Second one was mongodb. Sort of fixed that, but haven't actually managed to get the DB engine working again [but at least it's not eating the disk].
This time it is /usr/local/zenarmor/log/active/worker0 logging at about 1MBps. It's mostly lines like this:

2025-03-10T10:01:53.378794 WARN ArrayStream was full, str: osver pos: 8191
2025-03-10T10:01:53.378802 WARN ArrayStream was full, str: ":" pos: 8191
2025-03-10T10:01:53.378811 WARN ArrayStream was full, str: "} pos: 8191
2025-03-10T10:01:53.378820 WARN ArrayStream was full, str: ," pos: 8191
2025-03-10T10:01:53.378828 WARN ArrayStream was full, str: remote_device pos: 8191
2025-03-10T10:01:53.378847 WARN ArrayStream was full, str: ":" pos: 8191
2025-03-10T10:01:53.378855 WARN ArrayStream was full, str: " pos: 8191
2025-03-10T10:01:53.378864 WARN ArrayStream was full, str: ," pos: 8191
2025-03-10T10:01:53.378871 WARN ArrayStream was full, str: community_id pos: 8191
2025-03-10T10:01:53.378879 WARN ArrayStream was full, str: ":" pos: 8191
2025-03-10T10:01:53.378892 WARN ArrayStream was full, str: 1:NLGH3mmPeENTT1aXwYWl5XLicUw= pos: 8191
2025-03-10T10:01:53.378908 WARN ArrayStream was full, str: " pos: 8191
2025-03-10T10:01:53.378916 WARN ArrayStream was full, str: ," pos: 8191
2025-03-10T10:01:53.378924 WARN ArrayStream was full, str: handshake_result pos: 8191
2025-03-10T10:01:53.378931 WARN ArrayStream was full, str: ":" pos: 8191
2025-03-10T10:01:53.378939 WARN ArrayStream was full, str: None pos: 8191
2025-03-10T10:01:53.378964 WARN ArrayStream was full, str: " pos: 8191
2025-03-10T10:01:53.378972 WARN ArrayStream was full, str: }

What is it complaining about here?

In this dialogue we can see that we start on 500 and jump to 4500.
In particular this line: "parsed IKE_SA_INIT response 0 [ SA KE No V V N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) V ]" is this the far end telling me that NAT has been detected?  Because the next packet I send is sent to/from 4500. But I am sure no NAT is in use here.

Code Select Expand

14[ENC1] <the uuid|1> generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
14[NET1] <the uuid|1> sending packet: from 203.0.113.158[500] to 109.104.97.188[500] (336 bytes)
03[NET2] sending packet: from 203.0.113.158[500] to 198.51.100.188[500]
02[NET2] received packet: from 198.51.100.188[500] to 203.0.113.158[500]
02[NET2] waiting for data on sockets
12[NET1] <the uuid|1> received packet: from 198.51.100.188[500] to 149.106.180.158[500] (446 bytes)
12[ENC1] <the uuid|1> parsed IKE_SA_INIT response 0 [ SA KE No V V N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) V ]
12[IKE1] <the uuid|1> received Cisco Delete Reason vendor ID
12[IKE1] <the uuid|1> received Cisco Copyright (c) 2009 vendor ID
12[IKE1] <the uuid|1> received FRAGMENTATION vendor ID
12[IKE2] <the uuid|1> received FRAGMENTATION_SUPPORTED notify
12[CFG1] <the uuid|1> selected proposal: IKE:AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
12[IKE1] <the uuid|1> local host is behind NAT, sending keep alives
12[IKE2] <the uuid|1> reinitiating already active tasks
12[IKE2] <the uuid|1>   IKE_CERT_PRE task
12[IKE2] <the uuid|1>   IKE_AUTH task
12[IKE1] <the uuid|1> authentication of '203.0.113.158' (myself) with pre-shared key
12[IKE2] <the uuid|1> successfully created shared key MAC
12[IKE0] <the uuid|1> establishing CHILD_SA c942748f-a0ff-403f-8539-5a2fc2ba54f2{2}
12[ENC1] <the uuid|1> generating IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH N(ESP_TFC_PAD_N) SA TSi TSr N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
12[NET1] <the uuid|1> sending packet: from 203.0.113.158[4500] to 109.104.97.188[4500] (268 bytes)
03[NET2] sending packet: from 203.0.113.158[4500] to 198.51.100.188[4500]
02[NET2] received packet: from 198.51.100.188[4500] to 203.0.113.158[4500]

I had an annoying issue where a tunnel worked for 48h and then it just "gave up" even though there was traffic trying to flow from both sides [other end is a Cisco ASA but I don't manage it so can't say much more than that].
I set "Start action: Trap+Start" on both child SAs and that seems to have helped, has been up for nearly 90h now.

How is

Code Select Expand

local host is behind NAT, sending keep alives
determined? Is it due to what the far-end says ["you are behind NAT"], or is it some other heuristic? I am seeing it in a scenario where there is definitely no NAT at my end ["local host"] and almost certainly not at the far end.