Messages

I use production - needed for backup.
But I also tried Standard - just made a standard snapshot by hand and removed it again.

Both production as standard snapshot causes the errors to be displayed in the OpnSense VM.
No difference that I can see.

Wait a minute.
I just disabled the option "Create standard checkpoints if the guest does not support creation of production checkpoints".
And now it can not create a checkpoint anymore.

This means Standard checkpoints are used - Production checkpoints are somehow not supported by the OpnSense VM.

I use production - needed for backup.
But I also tried Standard - just made a standard snapshot by hand and removed it again.

Both production as standard snapshot causes the errors to be displayed in the OpnSense VM.
No difference that I can see.

[at some point there is a SCSI Status Error during a WRITE operation and PFSense/FreeBSD will become locked up or partially working but eventually will not respond to network or UI requests.]

Nothing that I can find.

But in the link, comment 6, is stated:

Code Select Expand

thomaslauer 2019-05-20 19:05:15 UTC

Hi, i have 120 PFSense VMs from 2.3.4 to 2.4.4-2 all Hyperv VMs with GEN2 and UFS.
and some vms with Hyperv GEN2 and UFS. All this VM has the same issue.

I have only one VM with GEN2 and ZFS. This VM has no SCSI Errors during the snapshot.


In comment 7 is stated:

Code Select Expand

Nick 2019-05-31 16:53:28 UTC

I am having this problem with PFSense (2.4.4-RELEASE-p3) running on Hyper-V (Windows 2012 R2). In my case, replication might run fine for a while (hours, days) but at some point there is a SCSI Status Error during a WRITE operation and PFSense/FreeBSD will become locked up or partially working but eventually will not respond to network or UI requests.  I would love a resolution to this.  For the moment, I've disabled replication and it's been fine.

Perhaps useful, perhaps not:  I've been running PFSense for years as a replicating VM on Hyper-V W2K2012R2 without issues.  It was just this week when I started having problems.  PFSense was previously running many different versions (2.2, 2.3, 2.4.3).  When I started having problems this week I had not upgraded PFSsense or the hypervisor.  As far as I can tell "nothing changed".

Nick

The problem is in comment 7: at some point there is a SCSI Status Error during a WRITE operation and PFSense/FreeBSD will become locked up or partially working but eventually will not respond to network or UI requests.

I hope the solution in comment 6: I have only one VM with GEN2 and ZFS. This VM has no SCSI Errors during the snapshot.
So, I am going to test this, and see what that brings - but in the past, I had other problems using ZFS ...

Just found this, which is exactly what I experience:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236042

Here exactly the same problem on Hyper V 2019 Gen 2
Hardware: HPE DL360 Gen10, 2x Xeon Silver 4210R, HPE Smart Array P408i-a SR Gen10 with battery.

Its not only the errors, it freezes about daily - especially when there is a lot of traffic.

I found a solution - or better: work around.

After the restart of the firewall, wait one or two minutes and then login, go to the Dashboard and restart the Packet Filter and the System Routing.
And then everything works again.

It started around februari 2025, suddenly, after a reboot of the firewall the NAT rules did not work anymore.
All rules have the logging on, but when I check in the live log, I don't even see any incoming connection, not accepted and not denied.
Other connections work as they should, only NAT doesn't work.

I checked, checked and checked again, removed contraints, recreated the rules, no avail.

But then: after about 4 hours (after the reboot) everything suddenly starts to work again.
Without me doing anything ....

And this is at EVERY reboot of the opnsense firewall.

Somebody? Any suggestions?

Found the setting, in case anyone needs it.

HTTP(s) Location – advanced settings – Advanced Proxy options – proxy Read timeout = 999999

A little bit late but:

OpenVPN instances, Miscellaneous, Options: persist-remote-ip

M@rch0n,

I have exactly the same problem using the NGINX proxy.
Without NGINX, no errors appear.

Did you ever find a solution?

On consumer internet contracts you won't be able to host your own DNS server which is open to internet, you need to either host that DNS on VPS like azure or AWS or setup VPN or proxy.

ISPs of most countries block incoming DNS traffic on UDP 53 to prevent people being able to mess up global DNS servers and DNS spoofing, outgoing smtp traffic on TCP 25 to prevent spamming and few other ports only hosting companies like google, amazon AWS, Microsoft and Eila Kaisla need, in fact some countries (for example Finland for one) even have laws which obligate ISPs to do that.

Its a business contract, not consumer.

But clear - port 25 incoming just works, but it look like 53 is blocked by the provider.
But I already fond another solution that solves the problem, and this can be closed as OpnSense is not the cause.

For the solution (in the case someone is interested):

Its for a branche office with 2 people - we do not have a site2site VPN and don't want one.
So I solved it by setting the DNS on the machines to 127.0.0.1 and add a NetSh Interface Proxy, listening on 127.0.0.1 port 53 and forwarding it to the destination IP on port 5353, that I NAT there to port 53 on the destination.
Not a perfect solution, but it works.

Unclear.
Do you mean you want to forward dns queries (port 53) from WAN to a specific machine on LAN , or within your LAN, or something else?

I want to forward DNS queries from WAN (originating from specific IP's) to a specific machine on LAN.

I am trying to MAP port 53 and some other ports, only from a specific source, and map it to a specific machine in the LAN.

All other ports work, but for 53, I don't even see the connections in the in the logs.
So, I don't see it even blocked.

Is this something from within OpnSense, or do I have to contact my Internet provider to ask him to open port 53?
Because, as far as I know, all ports should be open ...

What needs to be flushed to disk will be flushed to disk. And ZFS never does in-place overwrites. So if you write every 5 seconds or 18 times the transaction groups every 90 seconds ...

If @tverweij running virtualised instances is indeed using ZFS that specifically is a bad idea. Due to its copy on write nature you cannot thin provision virtual disks (well you can, but it doesn't make sense) because ZFS will eventually write every single disk block and so blow up the disk to its configured maximum size.

For virtual disks it's much better to use UFS and manage snapshots and backups at the hypervisor host level.

That is a good word of advice.
Maybe a good idea to add this advice to the installation docs?

Because I chose ZFS because the docs say:
"Install (UFS|ZFS) - Choose UFS or ZFS filesystem. ZFS is in most cases the best option as it is the most reliable option, but it does require enough capacity (a couple of gigabytes at least)."

A typical mSATA SSD used for embedded systems like the Transcend m370 series has got a TBW value of 180 for the 64 GB model.

At 120 kB/s write and using multiples of 1000 for calculation this means you can write to this disk for 1500000000 seconds = 416667 hours = 17361 days = 47.5 years before reaching the specified TBW.

Similarly your initial figure of 6 GB per day results in 30000 days or 82 years.

As I work on virtual machines (also for OpnSense), writes are always bad.
It is not only about wear of the SSDs used.

More important is that every write costs replication bandwidth, local backup space, cloud backup space and also causes the disks to wear.
9 GB (120kbs/sec) does not seem much, but translates to 9Gb traffic, 18Gb storage (replication), 18 Gb backups, 18 GB traffic for offsite backups and 18 GB of paid storage on the offsite backups. Per day.

Do This for 50 VM's and you find that a continuous write of 120 kB/sec is really a big waste of resources and adds a lot to the operational costs.