Topics

I am trying to set up so that all traffic on a VLAN gets routed out via a VPN client. I have read lots of tutorials and many posts, tried many different settings but with the instructions all being of different age they are intended for different versions of OPNSense and I am never sure which steps have changed or become redundant.

I have a OpenVPN client instance that is showing as connected, and I would like to link this to a VLAN I have already created. I get a bit lost in which steps are required manually and which are done by OPNSense automatically when it comes to

-interfaces
-devices
-gateways
-firewall aliases
-firewall NAT
-firewall rules
-DHCP

Is there a howto/tutorial based on the latest OPNSence describing how to do this? I run 25.7.2.

For ISP reasons, I am forced to connect my OPNSense router behind my ISP's fibre router, set to DMZ. When I compare the bandwidth I get directly from the fibre router to what I get through OPNSense, there is a significant reduction (although an improvement in bufferbloat). What could be the different contributing factors to this? In terms of hardware, the OPNSense router has:
CPU: AMD Embedded G series GX-412TC, 1 GHz quad Jaguar core with 64 bit and AES-NI support, 32K data + 32K instruction cache per core, shared 2MB L2 cache, 4GB DRAM
NICs: i210AT 1Gb/s

Fibre router:
You cannot view this attachment.

OPNSense behind fibre router:
You cannot view this attachment.

Are there specific OPNSense configuration tweaks I should try to reduce the bandwidth reduction?

I have been annoyed for some time that the graphs in the Reporting module use UTC time rather than the system time that is shown elsewhere, e.g. in the System Information gadget on the dashboard. Is there no way to get the Reporting module to respect the system time zone?

I searched for previous posts about this but only found this old thread from 2020.

If this needs to be an enhancement request, where do I post it?

When I upgraded from 23.1.7 to 23.1.9 a long time ago, my system started to show strong oscillations in CPU use, from 0 to 100, all the time. It would reach 100% about every 10 seconds or so. Because of the high average CPU, the temperature of the system went up by a couple of degrees. I have lived with this ever since because I couldn't isolate what was driving it.

After I upgraded from 25.1.7_2 to 25.1.7_4 and now 25.1.9_2, this oscillation has disappeared and now I am seeing a nice smooth line of CPU use (obviously subject to traffic volume). System temperatures are also lower.

Just thought I'd thank all the developers for continuous improvement - I have no idea which component makes the difference in my case, but this was a long-desired improvement.

I have a series of snapshots going back to 24.7.1. Their sizes vary greatly, and I would like to understand why and if there is anything I can do to influence the size.

I use the AdGuardHome plugin. Normally, I update it via its web interface. At the moment, there is an issue that has been fixed in the latest edge release but not yet released as a stable release. I know that you can update the plugin using the command line and -u, but I assume that only looks for a new stable release. Is there any way to install instead a specific release like the edge release?

After I upgraded from 24.1.10_8, my Health reporting has stopped working. See screenshot – in this case, the reporting worked for a while after the reboot but then stopped. After the next reboot, the reporting did not even capture anything. This is true of all the reports – the States reporting is just one example.

Also, the end time in the chart (in this case 16:55) never updates – after you have called up the graph the first time, the time remains the same.

Any pointers welcome on ways to track down the issue.

I have been running migumail's AdGuardHome plugin for a couple of years on OPNSense, without any issues. Then around the time of updates from 23.7.10_1 to the next version, I suddenly started having a strange issue. For most attempts to access a site, the first attempt fails (screenshot attached). When you refresh the web page, or when a device (such as one of my Squeezebox players) attempts a second time to access a stream, then it works.

Subsequent accesses in the same session (or on the same day, I haven't quite figured out the pattern) all work as expected.

The issue does not happen for all web sites; I have not yet figured out what determines when it works and when it does not.

The thing is, when I get one of these failures, if I then check the AdGuardHome query log, it shows neither a blocked nor a processed query request, so it is as if that first (and failed) request never reached AdGuardHome. The subsequent (successful) requests do show in the AdGUardHome query log.

This is why I am posting a question here rather than in the AdGuardHome forums – I do not think the problem lies with AdGuardHome, instead I suspect there must be some setting in OPNSense that is wrong (and probably changed with one of the upgrades). What can I do to investigate the issue?

In my log file System > Log files > Backend, I have thousands of entries like this:

Code Select Expand

Date SeverityProcess Line
2024-07-17T10:48:10 Error api no active session, user not found
2024-07-17T10:48:10 Error api no active session, user not found
2024-07-17T10:48:08 Error api no active session, user not found
2024-07-17T10:48:07 Error api no active session, user not found
2024-07-17T10:48:06 Error api no active session, user not found
2024-07-17T10:48:05 Error api no active session, user not found
2024-07-17T10:48:05 Error api no active session, user not found
2024-07-17T10:48:03 Error api no active session, user not found
2024-07-17T10:48:02 Error api no active session, user not found
2024-07-17T10:48:01 Error api no active session, user not found

Should I be concerned about this? Any way to diagnose and track down the cause?

We had a power cut. My APC UPS decided that this was the time to start beeping continuously, probably due to an old battery that needs to be replaced. After I reconnected power, my APU2E4 with OPNSense would not start properly. I connected via serial console and saw the following in the eternal boot loop :

Code Select Expand

>>> Invoking backup script 'netflow'
panic: ufs_dirbad: /: bad dir ino 6250145 at offset 512: mangled entry

I rebooted into single user mode and ran the following:

Code Select Expand

root@:/ # fsdb /dev/gpt/rootfs
** /dev/gpt/rootfs
Editing file system `/dev/gpt/rootfs'
Last Mounted on /mnt
current inode: directory
I=2 MODE=40755 SIZE=1024
        BTIME=Mar 22 08:21:31 2022 [0 nsec]
        MTIME=Mar 31 14:59:14 2024 [717610000 nsec]
        CTIME=Mar 31 14:59:14 2024 [717610000 nsec]
        ATIME=Dec 18 20:12:01 2023 [0 nsec]
OWNER=root GRP=wheel LINKCNT=22 FLAGS=0 BLKCNT=8 GEN=3eb55a60
fsdb (inum: 2)> inode 6250145
current inode: directory
I=6250145 MODE=40750 SIZE=1024
        BTIME=Nov 26 19:06:09 2022 [668628000 nsec]
        MTIME=Mar 31 12:25:14 2024 [721933000 nsec]
        CTIME=Mar 31 12:25:14 2024 [721933000 nsec]
        ATIME=Mar 31 12:25:10 2024 [0 nsec]
OWNER=root GRP=wheel LINKCNT=2 FLAGS=0 BLKCNT=8 GEN=2e8cf21a
fsdb (inum: 6250145)> clri 6250145
fsdb (inum: 6250145)> quit

***** FILE SYSTEM STILL DIRTY *****
*** FILE SYSTEM MARKED DIRTY
*** BE SURE TO RUN FSCK TO CLEAN UP ANY DAMAGE
*** IF IT IS MOUNTED, RE-MOUNT WITH -u -o reload
root@:/ # fsck -y -t ufs /dev/gpt/rootfs
** /dev/gpt/rootfs
** SU+J Recovering /dev/gpt/rootfs

USE JOURNAL? yes

** Reading 33554432 byte journal from inode 4.

RECOVER? yes

** Building recovery table.
** Resolving unreferenced inode list.
** Processing journal entries.

WRITE CHANGES? yes


***** FILE SYSTEM IS CLEAN *****
** 197 journal records in 22016 bytes for 28.63% utilization
** Freed 10 inodes (0 dirs) 9 blocks, and 11 frags.

***** FILE SYSTEM MARKED CLEAN *****
root@:/ # exit

Now the system is up and running, but I assume I still need to fix the following issue that was showing in the boot read-out on the console:

Code Select Expand

>>> Invoking backup script 'netflow'
./var/netflow/: Can't create 'var/netflow': No such file or directory
./var/netflow/metadata.sqlite: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/src_addr_details_086400.sqlite: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/src_addr_000300.sqlite: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/src_addr_003600.sqlite: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/src_addr_086400.sqlite: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/dst_port_000300.sqlite: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/dst_port_003600.sqlite: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/dst_port_086400.sqlite: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/interface_000030.sqlite: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/interface_000300.sqlite: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/interface_003600.sqlite: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/interface_086400.sqlite: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/src_addr_details_086400.sqlite-journal: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/src_addr_000300.sqlite-journal: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/src_addr_003600.sqlite-journal: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/src_addr_086400.sqlite-journal: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/dst_port_000300.sqlite-journal: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/dst_port_003600.sqlite-journal: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/dst_port_086400.sqlite-journal: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/interface_000030.sqlite-journal: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/interface_000300.sqlite-journal: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/interface_003600.sqlite-journal: Failed to create dir 'var/netflow': No such file or directory
./var/netflow/interface_086400.sqlite-journal: Failed to create dir 'var/netflow': No such file or directory
tar: Error exit delayed from previous errors.

What would be the best way to deal with this?

Ï used to have an XDSL connection that meant I had to use QoS to ensure my videoconferencing worked OK. When we were moved to fibre I just updated the bandwidth and quantum numbers in the traffic shaper settings and thought nothing more about it. Because of the much faster connection, things mostly work OK.

Then I decided to run a bufferbloat test and was surprised to see I only got a "C" rating from https://www.waveform.com/tools/bufferbloat. I connected via Ethernet directly to the OPNSense router and ran the test again - same result. Then I tried turning traffic shaping off and now I got a rating of "A". This puzzles me - why does the traffic shaping now make bufferbloat worse on my fibre connection, when it used to make things better on the XDSL connection?

My traffic shaping settings are:
Pipes
Down
  Bandwidth 430 Mbps
  CoDel not enabled
  (FQ-)CoDel ECN enabled
  FQ-CoDel quantum 1290
Up
  Bandwidth 290 Mbps
  CoDel not enabled
  (FQ-)CoDel ECN enabled

Queues
Down
  Weight 100
  mask destination
  (FQ-)CoDel ECN enabled
Up
  Weight 100
  mask source
  (FQ-)CoDel ECN enabled
Rules
Down
  Interface WAN
  Protocol ip
  Source any
  Src-port any
  Destination 192.168.2.0/24 (the OPNSense is 192.168.2.0 and hands out DHCP addresses 192.168.2.1-255)
  Direction both
  Target download queue
Up
  Interface WAN
  Protocol ip
  Source 192.168.2.0/24
  Src-port any
  Destination any
  Direction both
  Target upload queue

I could just leave the traffic shaping turned off and forget about it, but I would like to understand what is going on in case I need to turn it on in the future.

[connected directly to my ISP's fibre router]

I need help understanding an issue I seem to have with packet loss on my OPNSense router or my ISP fibre gateway.

I have a home network that connects to an OPNSense APU router, which in turn is connected to my ISP's fibre router (see diagram). The ISP router and the APU2E4 both have Gigabit NICs (in the case of the OPNSense APU, Intel I210-AT) and are connected by a brand new CAT6 Ethernet cable.



I was noticing that OPNSense was telling me about packet loss on the gateway, sometimes quite high (30%).



I decided to check the Quality/gateway chart and was shocked to see loss peaks of 50%. Comparing these in time to the traffic on the router, I can tell that the loss percentage goes up when there is little traffic and down when traffic is high, which I can see makes sense.



I am obviously not happy with having loss of this magnitude in the first place. I therefore installed smokeping on a Raspberry Pi and hooked it up to my network. First, I had it connected directly to my ISP's fibre router; then it was not showing any loss at all on Google, BBC, the ISP's DNS server, etc. Only one site showed some loss. Then I moved it and connected it behind the OPNSense router. There, I set smokeping to track some internet servers, the second and first hop of my ISP, the fibre router and the OPNSense router. Below are the results.









There is no indication of a problem behind the OPNSense router, but some internet servers are showing high loss numbers. The gateway (my ISP's fibre router) shows a very minor max loss. Strangely, there is a significant difference in max loss between the first and second hop (as determined by tracert) of my ISP - 62% and 8%, respectively.

At this point, my evidence is ambiguous - there seems to be something with my OPNSense router causing the packet loss, but how can I track this down? I am running AdGuard Home on the router, but it seems odd that this would generate packet loss as measured against the gateway by OPNSense. Also, I intentionally included both a web address version and an IP version of some servers for smokeping to test; for one server, the IP version got slightly less loss, for the other, slightly more. This suugests there is no consistent detrimental impact on packet loss by AdGuard. I also run traffic shaping, set up very simply with an upload queue and a download queue, set at the normal bandwidth delivered by my ISP (as measured by nightly speedtests from OPNSense). The shaping uses FlowQueue-CoDel ECN.

Any suggestions on how to further diagnose this would be very welcome.

Having just upgraded to 23.7.9, I am seeing much higher CPU usage than before, and the unit runs much warmer. While it isn't getting critical, I would like to understand what has changed and if there are parameters I should change or something I should fix.

Before the upgrade, the CPU usage would stay below 10% all the time except when there was heavy traffic such as a download or HD video streaming. Now, even when there is no traffic, it oscillates between 0% and 100%:


The CPU used to run at 61 degrees, now it is at 65.

Where should I start looking for the cause?

I am a relative newcomer to OPNSense. I use it in a home network setting and have modest hardware (APU2E4 with 4-core AMD GX-412TC SOC, 4GB RAM). I have a couple of questions relating to keeping an eye on traffic:

1. Are there any add-on solutions to improve the reporting/visibility of traffic, for example to see common web sites for outgoing traffic from a specific LAN IP? Any solution needs to either work on my modest hardware or on e.g. a RPi on the LAN, or on my QNAP x86 with Celeron 4-core J3455 and 8GB RAM. I have investigated ELK etc. but it seems these are too HW-demanding.
2. The other day my QNAP reported a suspicious connection attempt, even though I don't believe there should be a way for traffic from the WAN to get through the OPNSense FW. To check, I went to the OPNSense FW log file, plain view, and searched for the external IP of the suspicious attempt. That just left the interface saying "Loading..." forever. Initially, CPU use was quite high, but even after it had dropped back to ~5% the log file search screen still said "Loading...". The same happens if I search on an interface, such as wg1. Why is this?

I have recently implemented an OPNSense firewall and router for my home network. Looking at the built-in reporting available, I would like to see how I can get more comprehensive data and analysis of the traffic in/out, firewall actions, etc. Since my HW is limited, I suspect I'll set up the reporting on a separate machine, perhaps running the ELK stack or something similar. But, before I embark on this...

1. Are there more advanced reporting possibilities in OPNSense itself, perhaps with some added packages, even on modest hardware such as mine? I have seen mention on Routerperformance of Grafana, InfluxDB and other packages, but I am not sure if they would fit the bill.
2. If I go down the route of sending OPNSense data to an external reporting box, what would be a good way to start? I assume it should be possible to get some good data with what is already being generated on my OPNSense and without installing Zenarmor, right? Then should I go for pfELK, the integration from Elastic or something else?

When upgrading from 22.7.4 to 22.7.7, I got the following messages:

Code Select Expand

[71/74] Extracting os-wireguard-1.13_1: .......... done
Stopping configd...done
Starting configd.
Migrated OPNsense\Wireguard\Server from 0.0.3 to 0.0.4
Migrated OPNsense\Wireguard\Client from 0.0.6 to 0.0.7
Reloading plugin configuration

Fatal error: Uncaught Error: Class "phpseclib3\Crypt\Common\AsymmetricKey" not found in /usr/local/share/phpseclib/Crypt/RSA.php:69
Stack trace:
#0 /usr/local/etc/inc/certs.inc(34): require_once()
#1 /usr/local/etc/inc/config.inc(41): require_once('/usr/local/etc/...')
#2 /usr/local/etc/rc.configure_plugins(35): require_once('/usr/local/etc/...')
#3 {main}
  thrown in /usr/local/share/phpseclib/Crypt/RSA.php on line 69
Reloading template OPNsense/Wireguard: configd socket missing (@/var/run/configd.socket)
pkg-static: POST-INSTALL script failed

and

Code Select Expand

[72/74] Extracting os-ddclient-1.9_1: .......... done
configd not running? (check /var/run/configd.pid).
Starting configd.
Unable to lock on the pidfile.
/usr/local/etc/rc.d/configd: WARNING: failed to start configd
Migrated OPNsense\DynDNS\DynDNS from 1.4.0 to 1.5.0
Reloading plugin configuration

Fatal error: Uncaught Error: Class "phpseclib3\Crypt\Common\AsymmetricKey" not found in /usr/local/share/phpseclib/Crypt/RSA.php:69
Stack trace:
#0 /usr/local/etc/inc/certs.inc(34): require_once()
#1 /usr/local/etc/inc/config.inc(41): require_once('/usr/local/etc/...')
#2 /usr/local/etc/rc.configure_plugins(35): require_once('/usr/local/etc/...')
#3 {main}
  thrown in /usr/local/share/phpseclib/Crypt/RSA.php on line 69

Should I worry about these? Do I need to make any manual adjustments to my config?

I am confused by the manual/howto, specifically the text under "Choosing an interface" that mentions NAT and the WAN interface.

Because my ISP's router can't be put in bridge mode, I have set it to regard my OPNSense router as DMZ, so all traffic gets passed through to the OPNSense router and the OPNSense WAN interface gets an IP address on the ISP router. In this situation, can I get proper benefit from activating Suricata on the WAN interface to catch and stop intrusion attempts, or is this not possible?