Topics

Currently Wireguard has no CARP awareness meaning a few things:

* Wireguard that is stopped on a firewall that is BACKUP start after an XMLRPC config sync when the enable flag is set.
* Wireguard starts on a firewall that is BACKUP after a reboot.

This causes issues where two firewalls are connected to the same VPN endpoint and thus have the same tunnel IP. The far end thrashes packets all over as the two tunnels fight for dominance.

There is a good CARP script out there to deal with failover but not the XMLRPC sync issue.

It occurs to me that there should be an option within the WG plugin to make it "CARP aware" OR a change in the default behavior to always be CARP aware similar to DHCP.

My ask here is does a pattern for such a thing exist? Are there other plugins that have such functionality? I'd like to dig into the problem, but I want to use any established patterns if they exist so I don't make a pull request and find "hey dude, do it this way over here as that's how we always do it." I don't think any plugins ARE CARP aware, thus my question and perhaps a new pattern would need to be created. Thanks!

I am wondering how I can pin Wireguard to a gateway and/or cause it to fail over in a multi-WAN setup. This is only for outbound considerations as inbound - create the NAT and off I go.

Running 22.1.2_1. Seemingly Unbound never deletes/cleans up A records created by DHCP. As an example and how I can reproduce it - I spin up a new VM and it uses DHCP in its initial boot. Once it boots the first time, I create a static reservation and refresh the lease on the VM to use the new IP reservation. Now Unbound reports the original IP and the new reservation IP. Fast forward a few days and the original DHCP lease which has long expired literally days ago is still being served by Unbound along with the reservation IP. No amount of service restarts or reboots cleans it up. Is this a bug? By design and I am missing a config option? Something else?

EDIT: Ignore me... somehow the default lease time was set to a WEEK. Well, that explains that. It was a DHCP issue and not Unbound as when I deleted the entries from /var/unbound/dhcpleases.conf, they would reappear when I restarted Unbound. That was my clue. I r dum.

I have an odd problem where my WAN MTU gets set to weird values like 596 even though the interface is hard coded to 1500. While I try to track down what the root cause is I want to monitor the interface's MTU. The API seems very limited in this regard or I am not understanding something fundamental. I know the API works as I use it to enable/disable aliases and some other firewall rules. Any hints or better ways to get this value out of the box?

curl -k -u "user":"key" 'https://someip/api/interfaces/vxlan_settings/get'   

As well as /loopback_settings/get and getItem give me 401s. I figure I need to query and find a UUID to then use the UUID to get current state parameters as I do with other items. Any thoughts?

See my screenshot here: https://imgur.com/a/GfhtIB0

The question is why don't the two graphs correlate? If one looks at 10:40:55 you see there is a top talker using 4Mb yet the aggregate graph at the top shows only 620Kb. This makes no sense to me. I would expect the top aggregate graph to be at least 4Mb and likely more when considering other hosts talking at this same time over and above the top talker.

Unbound is spinning into space with crazy high CPU about once every hour. Kill -9 kills the process and the new process is perfectly happy for a bit until it too spins into space eating a ton of CPU and becoming unresponsive. The release notes don't show changes to Unbound, but this started happening after I updated this AM.

I'm dumb, Mods, please delete

Running 21.1.8 VNStat plugin 1.2_1

It seems I can only select 4 interfaces max or no statistics are listed. As soon as a I select a 5th interface, no stats are shown.

Here is example output:

Code Select Expand

vtnet3+vtnet2+vtnet1+vtnet0  /  hourly

         hour        rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     07/24/21
         15:00     19.76 MiB |   16.52 MiB |   36.29 MiB |   96.48 kbit/s
     ------------------------+-------------+-------------+---------------

Then as soon as I add vtnet4 I get

Code Select Expand

Error: Not all requested interfaces found in database or given interfaces aren't unique.

If I remove vtnet3 and keep vtnet4 I am back in business.

Code Select Expand

vtnet4+vtnet2+vtnet1+vtnet0  /  hourly

         hour        rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     07/24/21
         15:00     29.85 MiB |   25.44 MiB |   55.30 MiB |  143.12 kbit/s
     ------------------------+-------------+-------------+---------------


If I run iflist I get:

Code Select Expand

vnstat --iflist
Available interfaces: vtnet0 vtnet1 vtnet2 vtnet3 vtnet4 vtnet5 vtnet6 enc0 pflog0 pfsync0 ovpns1 wg0 wg1 wg2 wg3


This leads me to believe that the interfaces are seen by vnstat and plenty unique. I also recall in 21.1.7 (I think, or did I imagine this?) it broke out stats by interface where as now it appears it is showing a giant aggregate?

Then this shows stats per interface, so it seems it can collect stats per interface.

Code Select Expand

vnstat

                      rx      /      tx      /     total    /   estimated
enc0:
       Jul '21           0 B  /         0 B  /         0 B  /     --     
         today           0 B  /         0 B  /         0 B  /     --     

ovpns1:
       Jul '21           0 B  /         0 B  /         0 B  /     --     
         today           0 B  /         0 B  /         0 B  /     --     

pflog0:
       Jul '21           0 B  /    1.06 MiB  /    1.06 MiB  /     --     
         today           0 B  /    1.06 MiB  /    1.06 MiB  /     --     

pfsync0:
       Jul '21      2.05 MiB  /   10.71 MiB  /   12.76 MiB  /   15.33 MiB
         today      2.05 MiB  /   10.71 MiB  /   12.76 MiB  /   19.17 MiB

vtnet0:
       Jul '21     10.60 MiB  /   19.16 MiB  /   29.76 MiB  /   35.76 MiB
         today     10.60 MiB  /   19.16 MiB  /   29.76 MiB  /   44.73 MiB

vtnet1:
       Jul '21    208.17 KiB  /  635.98 KiB  /  844.14 KiB  /     --     
         today    208.17 KiB  /  635.98 KiB  /  844.14 KiB  /    1.24 MiB

vtnet2:
       Jul '21     15.84 MiB  /    2.97 MiB  /   18.81 MiB  /   22.99 MiB
         today     15.84 MiB  /    2.97 MiB  /   18.81 MiB  /   28.28 MiB

vtnet3:
       Jul '21           0 B  /  111.36 KiB  /  111.36 KiB  /     --     
         today           0 B  /  111.36 KiB  /  111.36 KiB  /     --     

vtnet4:
       Jul '21      7.65 MiB  /    6.72 MiB  /   14.36 MiB  /   15.33 MiB
         today      7.65 MiB  /    6.72 MiB  /   14.36 MiB  /   21.59 MiB

vtnet5:
       Jul '21      7.75 MiB  /    6.57 MiB  /   14.31 MiB  /   15.33 MiB
         today      7.75 MiB  /    6.57 MiB  /   14.31 MiB  /   21.51 MiB

vtnet6:
       Jul '21      2.12 MiB  /   10.86 MiB  /   12.98 MiB  /   15.33 MiB
         today      2.12 MiB  /   10.86 MiB  /   12.98 MiB  /   19.51 MiB

wg0:
       Jul '21    642.51 KiB  /    2.29 MiB  /    2.92 MiB  /    2.55 MiB
         today    642.51 KiB  /    2.29 MiB  /    2.92 MiB  /    4.38 MiB

wg1:
       Jul '21           0 B  /    4.69 KiB  /    4.69 KiB  /     --     
         today           0 B  /    4.69 KiB  /    4.69 KiB  /     --     

wg2:
       Jul '21      2.24 KiB  /    2.05 KiB  /    4.29 KiB  /     --     
         today      2.24 KiB  /    2.05 KiB  /    4.29 KiB  /       6 KiB

wg3:
       Jul '21           0 B  /         0 B  /         0 B  /     --     
         today           0 B  /         0 B  /         0 B  /     --   

And I can get individual stats via the CLI

Code Select Expand

root@OPNsense1:/var/log # vnstat -h -i vtnet0

vtnet0  /  hourly

         hour        rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     07/24/21
         15:00     11.35 MiB |   23.50 MiB |   34.84 MiB |   81.19 kbit/s
         16:00      1.77 MiB |    2.40 MiB |    4.17 MiB |  116.51 kbit/s
     ------------------------+-------------+-------------+---------------
root@OPNsense1:/var/log # vnstat -h -i vtnet1

vtnet1  /  hourly

         hour        rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     07/24/21
         15:00    214.40 KiB |  646.82 KiB |  861.21 KiB |    1.96 kbit/s
         16:00     28.98 KiB |   41.52 KiB |   70.50 KiB |    1.93 kbit/s
     ------------------------+-------------+-------------+---------------
root@OPNsense1:/var/log # vnstat -h -i vtnet2

vtnet2  /  hourly

         hour        rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     07/24/21
         15:00     20.10 MiB |    3.32 MiB |   23.42 MiB |   54.56 kbit/s
         16:00      2.01 MiB |  524.48 KiB |    2.52 MiB |   70.39 kbit/s
     ------------------------+-------------+-------------+---------------
root@OPNsense1:/var/log # vnstat -h -i vtnet3

vtnet3  /  hourly

         hour        rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     07/24/21
         15:00           0 B |  117.51 KiB |  117.51 KiB |      267 bit/s
         16:00           0 B |   20.10 KiB |   20.10 KiB |      548 bit/s
     ------------------------+-------------+-------------+---------------
root@OPNsense1:/var/log # vnstat -h -i vtnet4

vtnet4  /  hourly

         hour        rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     07/24/21
         15:00      8.05 MiB |    7.07 MiB |   15.13 MiB |   35.25 kbit/s
         16:00      1.39 MiB |    1.22 MiB |    2.61 MiB |   72.84 kbit/s
     ------------------------+-------------+-------------+---------------
root@OPNsense1:/var/log # vnstat -h -i vtnet5

vtnet5  /  hourly

         hour        rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     07/24/21
         15:00      8.16 MiB |    6.93 MiB |   15.08 MiB |   35.15 kbit/s
         16:00      1.44 MiB |    1.13 MiB |    2.57 MiB |   71.90 kbit/s
     ------------------------+-------------+-------------+---------------


It appears the default is to log to the syslog, but nothing seems to be in the syslog regarding vnstat.

So I don't know what to think or how to troubleshoot - nor have I seen any mention of there being a 4 interface limit.  Any thoughts?

EDIT: More thoughts...

vnstat --dbiflist shows all interfaces

Code Select Expand

vnstat --dbiflist
Interfaces in database: enc0 ovpns1 pflog0 pfsync0 vtnet0 vtnet1 vtnet2 vtnet3 vtnet4 vtnet5 vtnet6 wg0 wg1 wg2 wg3


This leads me to believe that the interface selection on the General tab is a configuration for reporting NOT a configuration on what to monitor - despite the face the help says "Set the interface to listen on.". So maybe I just don't understand how this plugin is supposed to work.

I have several apps that fail to setup video. Audio typically works. What'sApp and FaceTime are the main offenders. Zoom seems to be fine. Has anyone had experience with this? Any fixes? This feels like a similar issue to SIP back in the day where the media portion of the call is not setting up properly as there is no proxy. But every consumer cheapo router works with these protocols, so I am sure OPNSense can as well. I'm on 21.1.8 FWIW. Thoughts?

As the title says - the Updates tab no longer gives terminal output and just spins after a 21.1.7_1 upgrade. Are others seeing this or is this just me? Otherwise it seems packages install properly - it's just that now it's a black box and the first time I got a tad worried. If it's just me, ignore me. If it's not just me, I'll get a bug report in as Git doesn't show this as a reported issue.

EDIT: Feel free to delete this thread. Firefox at it again. I saw so many issues where a FF cache clear fixed things and what do you know, I cleared the cache and everything is fine. Sigh.

I have several apps that seem to not be able to setup video - namely meet.google.com and WhatsApp. FaceTime works perfectly well. In sniffing around it appears that both of those apps are trying to setup a dynamic UDP port inbound via WebRTC or similar. Essentially this feels like the old VoIP and passive FTP issues of yore. Is there a STUN server one can use with OPNsense? Or is there a built in fix for this? Another fix I haven't considered?

1:1 NAT is not an option as I have all sorts of random devices connecting on the LAN side and this just needs to work dynamically.

EDIT: I should mention, this is on 21.1.4

This is an embarrassing question - but where is the apply button? If you edit a rule, a blue apply button appears. Great. If you move on to another screen it disappears and for the life on me - I can't find where there is a master apply button.

I suppose I could toggle logging on/off for a random rule to make it reappear, but that seems hacky?

I want to automate a few items to toggle routes and and other simple things via the API. The first item I am trying to do is toggle enable/disable an alias. (Toggle alias on, a certain host in the alias matches a conditional route and routes out gateway B. Toggle off, it routes out gateway A.)

I have a user and API access is working. For instance, I ran

Code Select Expand

curl -k -u "sometoken":"somekey" https://192.168.40.2/api/firewall/alias/getAliasUUID/SomeAlias
and voila I got back

{"uuid":"44724741-c37e-419a-9fa2-3aad79111335"}

Fantastic. Auth works.

Now I am completely lost on making the POST. I tried this based on the POST example here https://docs.opnsense.org/development/how-tos/api.html.

Code Select Expand

curl -XPOST -d '{"ed8a5bf8-861a-4307-a7b4-df1b2a727d18":"null"}' -H "Content-Type: application/json" -k -u "sometoken":"somekey" https://192.168.40.2/api/firewall/alias/toggleItem

and I get

Code Select Expand

{"message":"action toggleItem expects at least 1 parameter(s)","status":400}

which I half expect as I have no clue what values to put in. The docs say Parameter $uuid,$enabled=null. Great, I have a uuid, what values are allowed for $enabled? I have tried true/false, 1/0. Is there a table somewhere that describes these values? Or am I so green there is some convention here that a real dev would understand that I don't? Thanks!

I have a working Wireguard tunnel to Mullvad - mostly. I have not set an MTU anywhere but the Wireguard interface shows an MTU of 496 which is bizarrely low. I cannot connect to most sites due to this low MTU and I have confirmed that that is indeed the MTU by running

Code Select Expand

ping -i 0.1 -D -g 300 -G 1500 8.8.8.8. Sure enough when I hit 497, pings die. I have tried setting the interface and the Wireguard Local MTU to 1420 (the usual default) and then no traffiic passes. I am stumped. Any thoughts on what's going on and how to get a normal/usable MTU?

This seems to be a pretty common topic, but I haven't found anything definitive. I have a DHCP address on my WAN. I have seen multiple work arounds involving spoofing MACs, using non-routable IPs on the WAN interface for CARP and others. It seems to me that simply doing an ifdown on the WAN interface of the backup firewall is fine for my use case.

The big question is, where should I create my notify logic? Can I do it directly in /usr/local/etc/devd/carp.conf or will that get overwritten with updates? Can I create another file /usr/local/etc/devd/mycustomtweaks.conf that will be safe from updates?

The title says it all. I can't seem to update the box and when I do I get the error "Could not authenticate the selected mirror."

I tried from the command line and got:

Code Select Expand

Fetching change log information, please wait... Certificate verification failed for /C=NL/ST=Zuid-Holland/L=Middelharnis/O=OPNsense
3472375263232:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
fetch: https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/sets/changelog.txz.sig: Authentication error

That sent up a red flag - that's the self signed certificate on the box?!

Then I tried curl just to see what's up:

Code Select Expand

*   Trying 89.149.211.205:443...
* Connected to pkg.opnsense.org (89.149.211.205) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /usr/local/etc/ssl/cert.pem
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self signed certificate
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

DNS seems to resolve to the right IP and a connection is made. https://imgur.com/a/XtQVxTN

I'm stumped. Any ideas?