Topics

Hi,

I have had some unstableness in my opnsense for over a year now. After long digging, I found it is likely caused by ARP jumping IP from device to another in my laptop. Why does this keep happening?

I have laptop with two interfaces, wlan and usbdongle ethernet when in wire:

Code Select Expand

2: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 6a:cb:3f:c6:c9:09 brd ff:ff:ff:ff:ff:ff permaddr 9c:67:d6:0f:8f:c0
    inet 192.168.117.59/24 brd 192.168.117.255 scope global dynamic noprefixroute wlp0s20f3
       valid_lft 4000sec preferred_lft 4000sec
    inet6 fe80::66f9:af89:6d28:703a/64 scope link tentative noprefixroute
       valid_lft forever preferred_lft forever
4: enp0s13f0u1u2u1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 48:65:ee:15:7f:c2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.117.56/24 brd 192.168.117.255 scope global dynamic noprefixroute enp0s13f0u1u2u1
       valid_lft 2936sec preferred_lft 2936sec
    inet6 fe80::4993:59f0:25d:240a/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

Both MACs are fixed with separate IP addresses in KEA reservations page:

192.168.117.0/24   192.168.117.56   48:65:ee:15:7f:c2   satechi
192.168.117.0/24   192.168.117.59   6a:cb:3f:c6:c9:09   iklap

Satechi is the usbdongle brand. Iklap is the Fedora laptop name.

While I have the both connected (docking), I see this bouncing in OPNSense:

Code Select Expand

arp: 192.168.117.56 moved from 6a:cb:3f:c6:c9:09 to 48:65:ee:15:7f:c2 on igb2
arp: 192.168.117.59 moved from 6a:cb:3f:c6:c9:09 to 48:65:ee:15:7f:c2 on igb2
arp: 192.168.117.56 moved from 48:65:ee:15:7f:c2 to 6a:cb:3f:c6:c9:09 on igb2
arp: 192.168.117.56 moved from 6a:cb:3f:c6:c9:09 to 48:65:ee:15:7f:c2 on igb2
arp: 192.168.117.56 moved from 6a:cb:3f:c6:c9:09 to 48:65:ee:15:7f:c2 on igb2

And I believe that will drain the opnsense out of mem soonish. What causes the IP to bounce outside of their mac? I suspect it's somehow the laptop sending dhcpc query with laptop name in it (NetworkManager), which then KEA uses to overrule what Reservations page is saying.

Is this a bug somewhere? Why does KEA allow ip to go from MAC to another not respecting reservations?

Any idea what should be done here? It's annoying needing to toggle wlan off each time while docking due this. Do I have some misconfig in a) in my Fedora laptop or b) KEA, or c) bug somewhere?

Hi,

I have been experiencing this for quite long, but would now get to the roots of it. I installed telegraf, influxdb and grafana to see when and what starts going wrong. I see flowd_aggregate.py script at least keeps using lot cpu. But I can't find from logs what causes sudden memory usage, and raises cpu usage too. See grafana:

I didn't know where to put the image, as I can't upload it here, but see from mastodon: https://mementomori.social/@ikkeT/113957621410576425

Code Select Expand

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND                                                                                                                                     
99283 root          1 120    0    51M    38M CPU0     0  46.7H  99.05% python3.11                                                                                                                                   


Code Select Expand

root@OPNsense:~ # ps awfux|grep 99283
root     99283  83.5  0.9   52676  39024  -  Rs   24Jan25  2804:23.00 /usr/local/bin/python3 /usr/local/opnsense/scripts/netflow/flowd_aggregate.py (python3.11)

Any ideas what could cause this, or how to find the problema from logs?

Hi,

I've had this problem for several months, but now getting more often. OPNsense works several days just fine, but all the sudden home traffic starts slowind down and then I can't access it any longer and network dies. I keep it up to date, it's nothing sudden, the problem has been around for several releases. Now I'm running 24.7.11.

I just had to pull the plug and reboot. I thought I look around a bit. I disabled rrd collection just to make sure it's not that. No help. I run the following services at home, not much traffic:
- HAproxy (mainly traffic to nextcloud instance
- dnsmasq for home gadgets
- kea dhcp
- captive portal for guest VLAN, hardly ever used.

I used to have IPv6 enabled, but after moving the new connection only has IPv4.

So not much running. Immediately I notice some problems:

1. Flowd is eating CPU:

Code Select Expand

76462 root          1 135    0    58M    44M CPU0     0  16:38 100.00% python3.11
# ps awfux|grep 76462
root   76462 100.0  1.1  59844 44944  -  Rs   09:23   16:57.09 /usr/local/bin/python3 /usr/local/opnsense/scripts/netflow/flowd_aggregate.py (python3.11)



2. Config.d Errors in logs

(I have never touched unbound, it's not running)

Code Select Expand

2024-12-18T09:44:55 Error configd.py [8741e584-e8e0-47d1-940e-639b0fe9a307] Script action failed with Command '/usr/local/opnsense/scripts/unbound/wrapper.py -s ' returned non-zero exit status 1. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/actions/script_output.py", line 78, in execute subprocess.check_call(script_command, env=self.config_environment, shell=True, File "/usr/local/lib/python3.11/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/opnsense/scripts/unbound/wrapper.py -s ' returned non-zero exit status 1.
2024-12-18T09:30:11 Error configd.py Timeout (120) executing : system diag log '20' '0' '' 'core' 'audit' 'Emergency,Alert,Critical,Error,Warning' '1734420490.461'
2024-12-18T08:55:33 Error configd.py [eb377147-ead9-4e22-b070-4066dc2a5e25] Script action failed with Command '/usr/local/opnsense/scripts/interfaces/list_macdb.py ' died with <Signals.SIGBUS: 10>. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/actions/script_output.py", line 78, in execute subprocess.check_call(script_command, env=self.config_environment, shell=True, File "/usr/local/lib/python3.11/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/opnsense/scripts/interfaces/list_macdb.py ' died with <Signals.SIGBUS: 10>.
2024-12-18T08:55:33 Error configd.py [47cd8873-4e90-45dd-81a7-66fa3dfee38c] Script action failed with Command '/usr/local/sbin/pluginctl -D ''' died with <Signals.SIGBUS: 10>. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/actions/script_output.py", line 78, in execute subprocess.check_call(script_command, env=self.config_environment, shell=True, File "/usr/local/lib/python3.11/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/sbin/pluginctl -D ''' died with <Signals.SIGBUS: 10>.
2024-12-18T08:53:14 Warning configd.py Stopping daemon.
2024-12-18T08:53:14 Error configd.py Configd disconnected while executing : interface list macdb
2024-12-18T08:52:52 Error configd.py Configd disconnected while executing : openvpn connections client,server
2024-12-18T08:52:52 Warning configd.py Stopping daemon.
2024-12-18T08:50:06 Error api no active session, user not found
2024-12-18T08:45:08 Error configd.py Timeout (120) executing : firmware remote
2024-12-18T08:43:06 Error configd.py Timeout (120) executing : firmware tiers
2024-12-18T08:41:28 Error configd.py Timeout (120) executing : firmware remote
2024-12-18T08:38:06 Error configd.py Timeout (120) executing : firmware remote
2024-12-18T08:38:05 Error configd.py Timeout (120) executing : firmware tiers
2024-12-18T08:36:05 Error configd.py Timeout (120) executing : firmware tiers
2024-12-18T08:33:04 Error configd.py Timeout (120) executing : firmware tiers
2024-12-18T08:23:11 Error configd.py Timeout (120) executing : firmware remote
2024-12-18T08:20:03 Error configd.py Timeout (120) executing : firmware tiers
2024-12-18T08:16:03 Error configd.py Timeout (120) executing : firmware tiers
2024-12-18T08:12:01 Error configd.py Timeout (120) executing : firmware tiers


3. Disk space should be OK

Code Select Expand

root@OPNsense:~ # ls -ltrh /var/crash && df -hT
total 4
-rw-r--r--  1 root wheel    5B Dec  2 21:45 minfree
Filesystem       Type     Size    Used   Avail Capacity  Mounted on
/dev/gpt/rootfs  ufs       13G    8.1G    4.3G    65%    /
devfs            devfs    1.0K      0B    1.0K     0%    /dev
tmpfs            tmpfs    2.0G    3.5M    2.0G     0%    /tmp
devfs            devfs    1.0K      0B    1.0K     0%    /var/dhcpd/dev
devfs            devfs    1.0K      0B    1.0K     0%    /var/captiveportal/zone0/dev


So question, what the heck is this flowd doing, and how to disable it? Perhaps it's that overcooking the CPU. I found some old thread about deleting and putting interfaces back to it, I'll try. Let's see what else is there.

Hi,

I got my first two wg peers to connect. But as I added the third peer, it won't get picked from gui to system configs. OPNsense is the latest version at the date, OPNsense 22.1.10-amd64. The config is just the same as the two previous ones, listing name, public key and allowed ip (192.168.116.22/24).

But when I apply the settings, only the two first ones are written into wg0 config file, also seen from gui in peer list. The third one never gets there. See Peer List view:

Code Select Expand

interface: wg1
  public key: (hidden)
  private key: (hidden)
  listening port: 55555

peer: (hidden)
  endpoint: 1.1.11.24:24472
  allowed ips: 192.168.116.21/32
  latest handshake: 10 minutes, 34 seconds ago
  transfer: 4.89 MiB received, 1.00 GiB sent

peer: (hidden)
  endpoint: 1.1.1.24:26682
  allowed ips: 192.168.116.20/32
  latest handshake: 36 minutes, 6 seconds ago
  transfer: 340.61 KiB received, 480.98 KiB sent


How can this be? I have tried saving and applying it several times, but the third one never get's there. Also restarted the wg several times. The config of peer is just like the others, only pubkey and ip are different. What am I missing?

Hi,

I've used this evening struggling with wireguard. I followed this guide [1] which tells to setup interface wg1 for the wg. There is some oddity in wg plugin, it doesn't list the first peer for example in list configs, which baffled me for a while, but I see from command line it's in config file.

However adding interface wg1, like instructed in guide, ruins networking. It seems to guide all traffic to this wg1. Which leads nowhere, naturally. Routing table looks like this, see the first line:

Code Select Expand


root@OPNsense:~ # netstat -r
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
0.0.0.0/1          link#9             US          wg1
default            87-92-128-1.rev.dn UGS        igb0


Which to me tells the wg1 grabs the traffic from LAN. As soon as I disable the wg1, the 0.0.0.0/1 line disappears, and the routing becomes normal and I get to internet from LAN.

Why is this, what did I do wrong?


[1] https://docs.opnsense.org/manual/how-tos/wireguard-client.html

Hi,

I was reading changelog for 22.7-rc1, and it says dh option gets dropped for openvpn. What needs to be done for client configs to make them being able to continue working?

Do I need to reconfigure all clients not to have dh option, and then update the box, and they somewhow get back connected?

As if I don't, I loose access to clients and they won't be able to connect back any longer after 22.7 update, right?

Hi,

I was thinking a way to get my home intranet service SSL certs automatically renewed. I don't like that all random stuff at home have self signed certs from some different dummy CA. I would like to make trust to OPNSense CA at home, and issue home certs from OPNSense.

But I don't want to do them manually. Is there an automated way for clients to ask for a new cert, and reinstall it when needed? Like what freeipa and certmonger does? Therr you define the cert in freeipa, and it keeps cert valid by automatically renewing it. Then clients use certmonger to keep the cert files updated, and services restarted when cert gets renewed.

Any similar method for OPNSense?

Hi,

I use haproxy for exposing my services to internet. I'd want to monitor the services behind the haproxy, and though it would be the easiest to use telegraf, which I also use, to publish the backend stats into influxdb. Grafana could then give alerts if they start failing.

Other options would be to run prometheus with alert manager somewhere, but I have all the above tools aready in place. Would telegraf plugin maintainer be interested of adding the haproxy endpoints to monitored objects? Has anyone already done that?

Or do you use some other alert mechanism for monitoring availability from OPNsense?

Hi,

I was going through GUI for options to set static IP for roadwarrior client. There is a fault in GUI guidance. The field in client config: IPv4 " VPN: OpenVPN: Client Specific Overrides: Tunnel Settings: Tunnel Network" states:

"This is the IPv4 virtual network used for private communications between this client and the server expressed using CIDR (eg. 10.0.8.0/24). The first network address is assumed to be the server address and the second network address will be assigned to the client virtual interface."

Sounds a lot like description for the variable that I'd need. However if I put there: "192.168.118.1 192.168.118.2" in order to get the server to use .1 and client .2 from the tunnel network, I get the error:

"The following input errors were detected:
The field 'IPv4 Tunnel Network' must contain a single valid ipv4 CIDR range."

So clearly the instruction in the GUI is wrong. I assume it would rather set the virtual client network CIDR, and the GUI is actually missing the "ifconfig-push" option that I'd need.

I would've created an issue about that, but I couldn't find the repo for this plugin.

I'd have a wish that this GUI misguidance was fixed, and the ifconfig-push option was added.

Hi,

any chance this little annoying bug would get fixed? See screenshot how the descriptions get encoded the web way, so special characters and spaces get en oded wrongly.

arkiaamua%252520edelt%2525E4v%2525E4%252520ilta

That should show "Arkiaamua edeltävä ilta", which means The night before weekday.

Hi,

I have a RasPi client which connects fine to openvpn in OPNSense. And raspi routes fine to my homenet. I'd like to route back to net behind raspi. OPNSense is the default gw at home.  How to do this?

I tried adding router into opnsense list of routes. Somehow it didn't work. How should be this done?

Let's assume home net is:
192.168.100.0/24, opnsense is 192.168.100.1

OpenVPN net is:
192.168.200.0/24, where both opnsense and client get addresses. Let's assume they would be .2 and .3. These are not static btw.

RasPi is in net:
192.168.300.0/24

How do I tell opnsense to route to 192.168.300.0 via 192.168.203?

Hi,

my certs won't t get renewed, and now I can't get new ones. It might be due having many HAproxy rules, perhaps one of them breaks acme.

Does anyone have idea where this loop fails at? What is it trying to do, and which might break it?

Code Select Expand


[Sat Aug  4 09:42:41 EEST 2018] ok, let's start to verify
[Sat Aug  4 09:42:41 EEST 2018] Verifying:mydomain.com
[Sat Aug  4 09:42:41 EEST 2018] d='mydomain.com'
[Sat Aug  4 09:42:41 EEST 2018] keyauthorization='snipped'
[Sat Aug  4 09:42:41 EEST 2018] uri='https://acme-v01.api.letsencrypt.org/acme/challenge/snipped'
[Sat Aug  4 09:42:41 EEST 2018] _currentRoot='/var/etc/acme-client/challenges'
[Sat Aug  4 09:42:41 EEST 2018] wellknown_path='/var/etc/acme-client/challenges/.well-known/acme-challenge'
[Sat Aug  4 09:42:41 EEST 2018] writing token:snipped to /var/etc/acme-client/challenges/.well-known/acme-challenge/snipped
[Sat Aug  4 09:42:41 EEST 2018] Changing owner/group of .well-known to root:wheel
[Sat Aug  4 09:42:41 EEST 2018] url='https://acme-v01.api.letsencrypt.org/acme/challenge/snipped'
[Sat Aug  4 09:42:41 EEST 2018] payload='{"resource": "challenge", "keyAuthorization": "snipped"}'
[Sat Aug  4 09:42:41 EEST 2018] POST
[Sat Aug  4 09:42:41 EEST 2018] _post_url='https://acme-v01.api.letsencrypt.org/acme/challenge/snipped
[Sat Aug  4 09:42:41 EEST 2018] _CURL='curl -L --silent --dump-header /var/etc/acme-client/home/http.header  -g '
[Sat Aug  4 09:42:42 EEST 2018] _ret='0'
[Sat Aug  4 09:42:42 EEST 2018] code='202'
[Sat Aug  4 09:42:42 EEST 2018] sleep 2 secs to verify
[Sat Aug  4 09:42:44 EEST 2018] checking
[Sat Aug  4 09:42:44 EEST 2018] GET
[Sat Aug  4 09:42:44 EEST 2018] url='https://acme-v01.api.letsencrypt.org/acme/challenge/snipped
[Sat Aug  4 09:42:44 EEST 2018] timeout=
[Sat Aug  4 09:42:44 EEST 2018] _CURL='curl -L --silent --dump-header /var/etc/acme-client/home/http.header  -g '
[Sat Aug  4 09:42:45 EEST 2018] ret='0'
[Sat Aug  4 09:42:45 EEST 2018] Pending
[Sat Aug  4 09:42:45 EEST 2018] sleep 2 secs to verify
[Sat Aug  4 09:42:47 EEST 2018] checking
[Sat Aug  4 09:42:47 EEST 2018] GET
[Sat Aug  4 09:42:47 EEST 2018] url='https://acme-v01.api.letsencrypt.org/acme/challenge/snipped'
[Sat Aug  4 09:42:47 EEST 2018] timeout=
[Sat Aug  4 09:42:47 EEST 2018] _CURL='curl -L --silent --dump-header /var/etc/acme-client/home/http.header  -g '
[Sat Aug  4 09:42:47 EEST 2018] ret='0'
[Sat Aug  4 09:42:47 EEST 2018] Pending
[Sat Aug  4 09:42:47 EEST 2018] sleep 2 secs to verify


It keeps doing that timeout loop. What is blocking it, any log which to follow for acme challenge?

Hi,

I just upgraded to 18.7 from the prior latest version. Now I've lost all my websites behind HAproxy. Postpone the update if you care about HAproxy. Unfortunately I am only with mobile, so I can't debug much. Rules have lost their names for example in frontend definitions, just uuids. Perhaps some broken config conversion.

I'll update after I get back to keyboard.

I took config backup before update. Can I try restoring it to the 18.7 box?

I really like how easy I can get certs now with HAproxy and Let's encrypt. Great work, thanks.

I thought I tell this minor issue with it here if someone is interested to fix it. It's really minor, but confusing for the first timer like me. So I have HAproxy listening all traffic to port 443. The default public service then routes requests to correct backends based on rules. First I create rules to sort traffic to given domain and backend. Then I add the ruleto to public service. That frontend also is using certs from Let's Encrypt service. Next I need to go to letsencrypt service to add domain.

I add the domain with http validation method, and press the small refresh button to force registration. This always fails at first run. But it also always works on the second run. Even though the gui don't show it until page refresh, but logs do.

So it's now OK for me, as I know this, but I sure spent some time on the first trials as I didn't look at the logs first.

Anyway, knowing this, it is great feature. Perhaps it gets fixed at some point.

Hi,

how does one do things like this in GUI? So replacing path parts with regexp placements?

reqirep  ^([^ :]*)\ /mirror/foo/(.*)     \1\ /\2


See sample here: https://www.haproxy.com/blog/howto-write-apache-proxypass-rules-in-haproxy/

Edit: describing a bit more:

So I want my external url to be rewritten by HAproxy for my internal server:

https:/my.com/myexturl/index.html?stuff=1
=>
http://foo.intranet/index.html?stuff=1

Like about this haproxy config:

    http-request set-header Host foo.intranet
    reqirep  ^([^ :]*)\ /myexturl/(.*)     \1\ /\2

    acl hdr_location res.hdr(Location) -m found
    rspirep ^Location:\ (https?://my.com(:[0-9]+)?)?/myexturl(/.*) Location:\ /\3 if hdr_location
    # ProxyPassReverseCookieDomain my.com foo.intranet
    acl hdr_set_cookie_dom res.hdr(Set-cookie) -m sub Domain= foo.intranet
    rspirep ^(Set-Cookie:.*)\ Domain=foo.intranet(.*) \1\ Domain=my.com\2 if hdr_set_cookie_dom
    # ProxyPassReverseCookieDomain / /myexturl/
    acl hdr_set_cookie_path res.hdr(Set-cookie) -m sub Path=
    rspirep ^(Set-Cookie:.*)\ Path=(.*) \1\ Path=/myexturl\2 if hdr_set_cookie_path

Now some of those I can find through menues, but e.g. the plain reqirep at the beginning I can't. How to do it via GUI?

Hi,

I'm asking for tips what's to be expected after upgrade gone bad with remote box.

I do have a OPNSense at remote location. Same HW as my home, apu2. At home I've successfully upgraded from 17 something to current every time there has been upgrade available. This remote box I tried to uograde yesterday for the first time in about half a year. I had ovpn on, and at first the uograde upgraded pkg. Then I suppose lot of 17 stuff. And a reboot I recall. Then the next upgrade required unlocking the 18 upgrade. I did.

After I then pressed the upgrade to 18 button, I got told it takes several reboots. I allowed it to go forward. The last message was about upgrading kernel and reboot. The next boot never brought up the OVPN or dynamicDNS update.

Luckily it's NATting as should, so people get to internet. I need to drive there having serial cable to see how to bring it back.

What do you think I'll face there, and what are the recovery actions?

If it's the worst, messed up install, will it work if I clean install 18 and import the 17 backup config?

Thumbs up reboot would fix it...

Hi,

I enabled transparent squid with adblocking, using UT1 list. It's great feature! However it didn't stop many Finnish ad sites, so I started digging into files and their formats. I noticed the urls or expressions don't get included, only domain list is added, and even that is formatted to include dot [.] in front of each domain. Is this just work in progress, or does opnsense squid not support the urls and expressions?

Another thing, what's the easiest way of adding my own list? Is it enough to have it in .gz file on own web server and adding it to remote control lists? Or have you noticed UT1 would take such input? It seems there is pretty good local list available for adblock plus: http://adb.juvander.net/Finland_adb.txt , which would be worth converting to squid format for myself, in case if it supported url and expressions files.

Here's how the file list looks like in UT1 block list for ads:

Code Select Expand

$ tree publicite/
publicite/
├── domains
├── expressions
├── urls
└── usage


Here's how expressions should work: http://www.squidguard.org/Doc/expressionlist.html

Hi,

I fancy a physical button at home, which would kill certain devices network connectivity. Think of kid not stopping playing after several mentions about dinner.... :)

So I could use my BT button which controls rules in my OpenHAB home automation box. That could then call API of OPNSense to toggle certain FW group on/off.

Is there such API, or any samples doing it e.g. using curl? I didn't find API in docs.

BR,
ikke

Hi,

I have been happily using VPN with OTP for a while. Now I would need to setup a VPN for some remote raspberry pi clients. I'd collect data from them over VPN. My question would be, can I add somehow exception for openvpn to not use one time password for those clients, but rather fixed passwords? Or just certs?

Like many web services have an option for "legacy" clients to have fixed passwords, like google talk clients.

-ikke

Hi,

is there any plugin to do dynamic dns with route53? I see on the list the dyndns plugin, but I'd like to use something else instead. I do this on one of my arm boxes: https://github.com/mthssdrbrg/ddns-route53

There seems to be different clients around, mainly using boto libraries. Has anyone yet happened to implement such plugin for OPNsense?